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Introduction 



This report summarizes the activities and findings of the external evaluation of the Science: It’s 
Elementary (SIE) program during the period from July 2008 through June 2009. In the 
program’s third year, the evaluation collected data from a variety of sources using several types 
of instruments: 

Questionnaires 

• Administered a questionnaire to collect baseline data on the third cohort of participating 
teachers and schools; 

• Administered a post-professional development questionnaire to gather teachers’ opinions 
of the quality and impact of the professional development; 

• Administered an end-of-year questionnaire to examine the impacts of the SIE program on 
teachers and their teaching; 

• Administered a questionnaire for principals of participating schools; 

Observations 

• Observed a sample of 28 professional development sessions to examine the quality of the 
training provided to teachers; 

• Conducted 12 classroom observations to examine the fidelity and quality with which 
teachers were implementing the SIE-provided modules; 

• Observed the June 2009 Eeadership Conference designed to prepare teacher leaders to 
provide professional development in future years of the program; 

Interviews 

• Interviewed a sample of 28 teachers participating in the SIE program; 

• Interviewed a sample of 27 principals whose schools are participating in the SIE 
program; ^ 

• Interviewed all district science supervisors/curriculum coordinators from the 14 Cohort 3 
districts; 

Assessments 

• Developed four student assessments, one for each grade 3-6 related to the science 
modules provided by the program for that grade; 

• Administered student assessments in the Eall of 2008 in schools participating in the 
program; and 

• Administered four assessments of teacher content knowledge at Module Enrichment 
workshops.^ 



* For the teacher and principal interviews, HRI drew a random initial sample and made repeated attempts to 
interview everyone in the sample. HRI randomly selected additional participants as needed to make up for non- 
response. In an attempt to interview 30 teachers (10 from each cohort), 65 teachers were contacted repeatedly via 
email and telephone. Thirty-six principals were contacted in an attempt to reach the targeted goal of 30. 

^ HRI received data from only 32 teachers across the four assessments administered at Module Enrichment 
workshops. This small number of responses prohibits any meaningful analyses of these data; thus, these data are not 
included in this report. 
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After a brief description of the SIE program and the evaluation plan, this report describes the 
teachers, schools, and districts participating in SIE; the nature and quality of the services 
provided by the program; and evidence of the program’s impact on teachers, their teaching, and 
their students. The report concludes with a summary of major achievements and HRI’s 
recommendations for the program. 
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Overview oe Science: It's Elementary 



The SIE program is managed by ASSET Inc. and overseen by the Pennsylvania Department of 
Education. SIE is an initiative aimed at improving elementary science instruction across the 
commonwealth of Pennsylvania. The program is focused on helping schools and districts 
implement an inquiry-based, hands-on science education program, with the ultimate goal of 
improving student learning. To accomplish its goals, SIE provides participating schools with 
module -based science instructional materials; teacher professional development around specific 
modules and inquiry -based science teaching more generally; and opportunities for strategic 
planning to help create supportive systems for science education reform. The program has six 
main components: 

1. Strategic Planning Institute: Based on the National Science Resource Center’s EASER 
model, these institutes are intended to help schools and districts understand, plan for, and 
successfully implement inquiry-based elementary science programs. Each school joining 
SIE sends a team comprised of administrators, teachers, and community partners to the 
Strategic Planning Institute (SPI) to learn about science education reform and to develop 
a three-year strategic plan for their school. 

2. Vision Conference: This one-day event for school, district, and community leaders is 
intended to help foster a common, long-term vision for science education among a larger 
and more diverse group of stakeholders than the SPI. Attendees include superintendents, 
district administrators, school board members, principals, and community representatives 
for both newly joining and prospective districts. 

3. Teacher Professional Development: SIE offers three types of professional development. 
Teachers new to SIE attend a one-day Eoundations course to learn about the program’s 
vision for science teaching. In addition, each teacher responsible for teaching an SIE- 
supported science module is expected to attend a one- or two-day Initial Module 
Training^ focused on that module. This training provides teachers with an opportunity to 
familiarize themselves with the instructional materials; receive practical tips for using 
them in their classrooms; and learn teaching strategies, such as integrating science and 
literacy through the use of science notebooks. After teachers have implemented a module 
twice, they are offered a two-day Module Enrichment Training on that module to learn 
more about the targeted science content and how to implement the module more 
effectively. Teachers repeat this professional development cycle of Initial Module 
Training and Module Enrichment for each SIE-supported module they implement. 

4. Delivery of Classroom Science Materials: In addition to the professional development, 
the SIE program delivers science modules directly to schools for teachers to use in their 
classrooms. After instruction with the modules is completed, they are returned to ASSET 
where they are refurbished for use again the following year. 



^ Most Initial Module Trainings are two days long; because some of the primary grades modules contain fewer 
lessons than the other modules, the Initial Module Trainings for these modules are one-day long. 
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5. Leadership Conferenee: This three-day eonferenee is intended to help develop the 
eapaeity of teaehers seleeted by sehools partieipating in SIE to beeome teaeher leaders 
who ean mentor and eoaeh other teaehers in their sehools and distriets. The program also 
expeets that some of these teaeher leaders will later be able to serve as faeilitators of 
Initial Module Trainings. 

6. Showease of Student Learning: To help build eommunity support for seienee edueation 
improvement, SIE eneourages eaeh partieipating sehool to host a Showease of Student 
Eearning. During this event, parents, and other eommunity stakeholders are engaged in 
seienee aetivities that highlight the hands-on, inquiry approaeh. These aetivities are 
typieally led by students who have experieneed this approaeh in their seienee instruetion. 
SIE provides materials and support to sehools for hosting these events. 

Eigure 1 shows the program’s theory of aetion, i.e., how program aetivities are intended to fit 
together to produee the desired short- and long-term outeomes. 
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Figure 1 



In addition, the SIE program requires each school to designate a Support On Site person (SOS) 
The SOS serves as the main conduit between the SIE program and the school, both for passing 
along important information from the program to the teachers and for helping teachers resolve 
any problems with the modules or professional development that may arise. The SOS is also 
responsible for registering teachers for SIE professional development and coordinating 
evaluation activities for the school. 
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In Year Three, SIE added another cohort of schools to the program. In total, over 2,500 teachers 
in 138 schools participated in the SIE program this year. Table 1 shows the distribution of 
schools across the education regions in Pennsylvania. 



Table 1 



Distribution of Schools Participating in SIE 





Number of Schools 


Region 


Total 


Cohort 1 


Cohort 2 


Cohort 3 


r 1 


18 


10 


8 


0 


East J 2 


12 


4 


3 


5 


1 3 


7 


4 


3 


0 


r 4 


14 


4 


4 


6 


Central J 5 


18 


9 


8 


1 


1 6 


23 


11 


10 


2 


f 7 


22 


9 


12 


1 


West J 










1 8 


24 


15 


8 


1 


Total 


138 


66 


56 


16 
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Overview of the SIE Evaluation 



The evaluation plan for SIE was developed by HRI in conjunction with key stakeholders with the 
goal of examining the major elements of the program’s theory of action. The questions driving 
the evaluation focus on four main areas: (1) the development of an aligned system to support 
effective science education; (2) professional development for teachers; (3) the impacts of the 
program on teachers and their teaching; (4) and the impacts of the program on students. The key 
evaluation questions, by area, are: 

System Alignment 

1 . How are school systems changing to support effective science instruction as a result of 
SIE? 

Professional Development 

2. What is the quality of the professional development provided to teachers? 

3. What is the quality of the SIE leadership training? 

Impacts on Teachers and Their Teaching 

4. What is the impact of SIE on teachers’ preparedness to implement the modules? 

5. What is the impact of SIE on teachers’ implementation of the modules? 

Impacts on Students 

6. What is the impact of SIE on student achievement in science? 

This report provides data to help answer these questions based upon the program’s activities in 
Year Three. 
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The Context eor Improving Science Instruction in SIE 

Schools 



Although helping schools and districts create fully-aligned systems is beyond the scope of SIE’s 
mandate, the school and district contexts in which the program is being implemented will likely 
factor into the success of the program. Data from teacher and principal questionnaires, as well as 
interviews with samples of teachers, principals and Cohort 3 district supervisors shed light on 
how these contexts are impacting SIE’s efforts to improve elementary science education. 



Baseline Status of Cohort 3 Schools 

Overall, the science programs of Cohort 3 schools (i.e., those joining SIE in Year Three) prior to 
participating in SIE were very similar to Cohort 1 and 2 schools when they joined the program. 
Interviews with random samples of Cohort 3 principals and teachers indicated that prior to SIE 
the majority of Cohort 3 schools’ science instruction was very traditional and uninspiring, 
relying heavily on lecture, textbooks, and worksheets. Typical responses included: 

Prior to joining SIE, I think we had the mindset, not intentionally, but I think our mindset 
was, “Let’s get our information out of a book.” [Principal] 

We were very textbook-based, very worksheet dependent.. .Pretty much lecture and taking 
notes and here is an activity on paper that you need to do. [Principal] 

We were using a very old science textbook, 20 years old. . . We would read with the book 
and perhaps provide a video to supplement it or bring in an experiment I could find 
online to do. Tests were paper and pencil. [Teacher] 

In addition. Cohort 3 teachers’ prior experience with kit-based science instruction was minimal. 
On a baseline questionnaire administered to teachers attending a Eoundations course (i.e., those 
teachers new to the SIE program), only about one-quarter of respondents indicated they had 
previously taught science using a module like those provided by SIE."* Of these teachers, fewer 
than half had received professional development on those modules. 

When asked to identify the strengths of their school’s science programs, approximately one fifth 
of baseline questionnaire respondents indicated the use of hands-on or inquiry-based instruction 
as the greatest strength. Teachers also cited aspects of their schools’ curriculum and the high 
quality of the staff as particular strengths, although each was mentioned by fewer than one- 
quarter of respondents. Other responses, given by only a handful of respondents apiece, included 
having a program that is interesting to students, and having high quality science textbooks. 
Sample responses included: 



"^The baseline questionnaire was administered at the beginning of each Foundations course by program staff. HRI 
received 448 baseline questionnaires. 
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The main strengths of my school’s science program are that it is hands-on with very little 
reading and memorizing involved. Through the hands-on activities, the students get 
excited about their role in experimentation, and it works for special education students 
whose reading level is not at grade level. 

Currently, we are teaching [science] through our reading program. The main strength is 
that we are incorporating science with reading. 

We have experienced teachers who plan and teach a strong standards-based curriculum. 

When asked how their school’s science programs could be improved, Cohort 3 teachers 
identified a number of areas. Approximately, one-quarter of baseline questionnaire respondents 
noted that their schools need more resources and/or instructional materials for science. The need 
to incorporate more hands-on/inquiry-based approaches to teaching was also mentioned by close 
to a quarter of respondents. In addition, over fifteen percent of respondents indicated that more 
instructional time was needed for science. Other areas for improvement mentioned, none by 
more than a few teachers, were elevating the emphasis on science to equal that of reading and 
mathematics, increasing alignment with state standards, and increasing the number of topics 
covered. The following comments illustrate teachers’ responses: 

My school’s science program has not been updated in many years. It could be improved 
with more up-to-date information, materials, and instructional methods. 

Our school’s science program can be improved by purchasing more materials and 
supplies relating to science. 

We need more hands-on experiences and inquiry-based instruction. 

Cohort 3 Schools’ Visions for their Science Programs 

In addition to characterizing the initial profile of their schools’ science programs prior to joining 
SIE, Cohort 3 teachers, principals, and district science supervisors were asked in interviews to 
describe their vision for their science programs. The overwhelming sentiment, among both 
teachers and administrators, was that they would like to see their science programs move toward 
a more hands-on, inquiry-based approach. This result is not surprising given that these schools 
chose to participate in the SIE program. Typical responses included: 

7 would love to have students doing science-related things every day, hands-on kinds of 
things. [Teacher] 

Our vision is to move from a textbook-driven, memorize the facts, teacher tells you what 
is going to happen and why, to an inquiry-based, hands-on doing science [program]. 
[District Supervisor] 

We are really looking at having an inquiry-based approach to teaching science. ..One of 
the reasons that we were so interested in being part of SIE is because that is the 
approach that the modules take. [Principal] 
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Many interviewees also indicated that they believed students would gain a better understanding 
of science and be able to apply that knowledge to real-world situations as a result of a more 
hands-on, inquiry-based program. About half of the interviewed district supervisors also 
indicated that they hoped having the SIE program in their school buildings would result in an 
increase in excitement about, and interest in, science, among both students and teachers. As two 
said: 



What I hope will happen will be that our faculty and our kids will get excited about 
learning and that our kids will learn those critical skills that they need to be successful in 
anything that they do. 

I am hoping that the students get excited about science and that they engage the parents 
in science as well, and it gets out to the community that we are participating in a really 
unique program that is going to help the students have more of an advantage to possibly 
select careers in science in the future. 

Role of SIE in Achieving Those Visions 

According to Cohort 3 district supervisors, the SIE program is an integral part of their plans for 
improving elementary science instruction. All 14 of the interviewees indicated that the SIE 
program has already been instrumental by providing professional development and instructional 
materials. As two supervisors stated: 

We knew we wanted to go to this more kit approach, hands-on approach... it [SIE] 
basically serves as the foundation of what we want to accomplish. 

Well, SIE has been very helpful in providing the professional development. . .and also 
providing the modules... in doing the professional development it will certainly help our 
teachers have more skill in the classroom to do inquiry-based instruction. 

Another aspect of the SIE program that district supervisors indicated has been particularly 
helpful in moving Cohort 3 schools towards realizing their vision for the science programs at 
their schools was the Strategic Planning Institute (SPI). Most indicated that the SPI provided 
them with an opportunity to develop a plan for integrating SIE into their schools. As one 
supervisor described: 

We were able to generate a great, very useful plan for us. When I came back, I actually 
shared it with the board and got them to approve it with the understanding that it is a 
working document and it is going to change as we work through the process. It really 
energized my staff, the ones that were able to go, and they came back and talked to their 
colleagues and that is great to have happen. There is kind of this excitement in the 
air... that helped the whole process. 

When asked how the SPI could have been improved, a few district supervisors indicated that 
they would have liked to bring more district stakeholders, and several said the SPI could have 
been abbreviated. 
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Changes in SIE Schools’ Science Programs 



In Year Three of the program, Cohort 3 schools were just beginning the process of improving 
their science programs through participation in SIE, while Cohort 1 and 2 schools were in their 
third and second year, respectively, of the process. In addition to Cohort 3 district supervisors, 
samples of teachers and administrators, representing all three cohorts, were asked in interviews 
to comment on the changes that have occurred in their schools since joining SIE. Nearly all 
Cohort 3 district supervisors, and all interviewed principals across the cohorts, indicated that 
science has become a higher priority in their schools as a result of participating in SIE. 
Representative comments included: 

I would say now [ science] is a very high priority. . .Before I would say that it was very low 
level... I don’t believe we really had science instruction to speak of that was part of our 
curriculum in K-2. . .Now we have a comprehensive K-5 program. [District Supervisor] 

Before, [science] wasn’t a high priority at all because the science curriculum was pretty 
poor, non-existent. ..It just got put on the back burner. ..Science was the least of our 
worries. . . What I have come to learn is the role science plays in everything, in reading, in 
social studies, in math. You could build your curriculum around science. ..It is such a 
good catalyst for getting [students] to be psyched about reading and writing and doing 
math because it is all related to these interesting explorations that they are doing. 
[Principal] 

Although science instruction has increased in priority in many SIE schools, about one-third of 
Cohort 3 district supervisors noted that science instruction is still not as high of a priority as 
reading and mathematics. As two indicated: 

7 don ’t think any Pennsylvania school can say [science] is number one when we are on 
the carpet for reading and math every year... certainly it is number two right now. 

[Science] would likely come in third; obviously reading and math have to be the priority 
right now in this state and country... reading comes first, then math, then science. 

One of the most telling indicators of the priority given to science is the amount of instructional 
time devoted to it. Based on teacher questionnaire data collected at the end of each year of the 
program,^ the amount of time devoted to science instruction in Cohorts 1 and 2 schools has 
increased in their second year of participation. (See Table 2.) The roughly three minute per day 
increase for Cohort 1 and the one-and-a-half minute per day increase for Cohort 2 are both 



^ In Year Three, the End-of-Year Questionnaire was administered via the Internet with the assistance of each 
participating school’s SOS. SOSs were asked to update a list of teachers at their schools, disseminate the URL for 
the survey, and follow-up with non-responding teachers. HRI received completed questionnaires from 1,773 of the 
2,3 1 1 teachers that were registered for the survey, a response rate of 77 percent. 
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relatively small effects (0.27 and 0.12 standard deviations, respectively).^ Across an entire year 
of instruction, this small amount of time adds up to an additional nine hours or so of instructional 
time in Cohort 1 schools and four- and- a-half hours of instructional time in Cohort 2 schools. 



Table 2 



Time Devoted to Science Instruction, by Years of Participation 





Number of Minutes per Day 


First Year 


Second Year 


Third Year 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Cohort 1 


14.23 


11.98 


17 . 62 * 


12.90 


17.85 


11.93 


Cohort 2 


15.83 


12.22 


17 . 34 * 


12.25 


— 


— 


Cohort 3 


14.73 


13.01 


— 


— 


— 


— 



* Instructional time significantly different than previous year, p < 0.05 (Cohort 1: one-way ANOVA with 
Tukey HSD post-hoc comparisons; Cohort 2: independent samples t-test). 



In some cases, schools have added science to their instructional schedule; in other cases schools 
have increased the instructional time scheduled for science. The following remarks were typical: 

In the past, science, you would take it whenever you could... now [teachers] have these 
kits and they know they will only have them for a certain length of time, so it kind of 
forces them to have science instruction every day. [Principal] 

[Teachers] are doing science. I would love to say they always did, it is in the 
schedules... but now they are making the time to do it because they see the children’s 
interest... [Science] was definitely not taught to the extent that it is taught now. [Principal] 

We used to have a very limited amount of time for science and now we have 40 minutes a 
day every day with SIE. [Teacher] 

There have been a number of other instructional changes in schools since they joined SIE. 

Nearly half of interviewed principals noted that teachers are covering science topics in greater 
depth, with the trade-off that they are covering fewer topics overall. These principals indicated 
that this change has been a positive one. As one principal explained: 

I would say that there are actually fewer science topics taught, but the science topics 
covered are being covered in more depth. So the students are learning more of the 
scientific principles and the scientific method even though they have fewer topics. 

The instructional strategies that teachers are using to teach science are also changing. Nearly all 
of the interviewed principals and over half of interviewed teachers indicated that their schools 



® When comparing two means, the effect size is calculated as the differences between the two means divided by the 
pooled standard deviation. Effect sizes of about 0.20 are typically considered small, 0.50 medium, and 0.80 large. 
Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence Erlbaum 
Associates. 
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have shifted science instruction to a more hands-on approach since their schools joined the SIE 
program. Representative comments included: 

[Science instruction] is not give, give, give all the information and then memorize it and 
take a test; it is, “Let’s discover together. ’’ [Principal] 

In the past it [science instruction] was pretty much read the book and answer the 
questions... Now [students] are actually using equipment; it is a hands-on approach, a 
discovery approach, and so it is much different. [Principal] 

It’s different now, because we are. ..doing an experiment every day. The hands-on with 
the children happens now, where before it was more textbook, or let’s read and talk, and 
less of them doing stuff. [Teacher] 

It’s all hands-on. I tried to do hands-on, but with time and everything else I could not do 
as much that I would like. Now every day or every other day they are having a hands-on 
experience through an experiment. [Teacher] 

The use of notebooks in science was also mentioned by several teachers as a particularly 
valuable strategy that they now use in their science instruction. As one teacher said: 

Using the notebook has changed my instruction. . .Actually giving them a focus question 
and having them really critically think about it and even write a response. I think it has 
led to a better understanding of the focus of the lesson, this is what we’re going to be 
learning, and then we come back to it at the end. So I think it makes it more formal, 
rather than just a class discussion or students sharing thoughts. 

When asked in interviews to identify impacts in their schools as a result of participating in SIE, 
several principals indicated that teachers are more enthusiastic about and confident in teaching 
science. Eor example: 

The excitement towards education, just as a profession, has increased. . . The kids are 
motivated, they get excited, they want to do it so it makes the teachers motivated and they 
are working harder to have [students] reflect and be challenged. 

I just feel that there is an enthusiasm and excitement about teaching science that through 
all of the training has been passed on to the teachers. 

The overwhelming sentiment among district and school administrators was that they appreciated 
the SIE program and found it to be very influential in effecting change in their science programs. 
They indicated that the instructional materials provided by the program were particularly 
valuable and that SIE was meeting a need for high-quality professional development in science. 
Comments included: 
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[The instructional materials] are very good... the teachers, many of them don’t have a 
background in science, they don ’tfeel comfortable teaching it, but these lessons allow 
them to feel comfortable and to do a good job with it. [Principal] 

If you don ’t have the professional development, you don ’t have a program quite 
honestly. ..If you don ’t have effective professional development it does not matter what 
piece of material you have, you have to have someone who is comfortable teaching the 
content. [District Supervisor] 

I have a very good opinion of [the SIE program]. I think that money talks and I think the 
state made a good investment in a program that delivered... not only quality materials, 
but quality professional development... Districts that have invested the time... are reaping 
the benefits of it... We have been very lucky to have participated in it over the last couple 
of years. [Principal] 

Administrators also shared that, in the past, a lack of funding for materials and professional 
development had limited the quality of their schools’ science programs. A few specifically 
indicated that, without the support of the SIE program, they would not be able to fund a similar 
program on their own. Several principals were concerned about their schools’ ability to sustain 
the effort given the uncertainty of continuing state funding. In the words of two principals: 

It is a very good program. I hope that the state legislators and the governor will be able 
to find the necessary funding. . .to allow the state and our students to be able to go 
through this for years to come. 

The only challenge .. .would be the funding issue... If the governor chooses not to fund the 
program, it will be difficult for our district to maintain. 



Factors Affecting Science Program Improvement 

On a questionnaire, principals from each cohort were asked about factors that facilitate or inhibit 

n 

their schools’ science programs. As can be seen in Table 3, nearly all responding principals 
indicated that state science curriculum frameworks and access to professional development for 
teachers were facilitating factors. The quality and availability of resources for science 
instruction (presumably those being provided by SIE) were also frequently seen as helpful. State 
and district testing in subjects other than science were the most frequently mentioned inhibitors, 
though each of these was selected by fewer than 1 in 5 principals. 



^ The Principal Questionnaire was a web-based survey administered to the principals of all SIE schools. HRI 
contacted principals via email, following up with additional emails to non-responders. Of the 135 principals (a few 
individuals were principals for multiple schools), 100 responded to the survey, a response rate of 74 percent. 
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Table 3 



Principal Opinions of Factors that Affect Science Instruction 





Percent of Responding Principals 
(N = 99) 


Facilitates^ 


Inhibits* 


State science curriculum frameworks 


92 


2 


Access to professional development for science 


91 


7 


Quality of resources for science instruction 


91 


4 


Availability of resources for science instruction 


89 


7 


District science curriculum frameworks 


88 


4 


Importance that the district places on science instruction 


86 


7 


State science testing policies and practices 


80 


8 


Consistency of science reform efforts with other district/school reforms 


80 


5 


District science testing policies and practices 


71 


4 


State testing policies and practices in subjects other than science 


62 


19 


District testing policies and practices in subjects other than science 


58 


12 


District grading policies and practices 


51 


5 


District policies for evaluating teachers 


51 


4 


District structures for recognizing and rewarding teachers 


27 


5 



Includes those indicating “Greatly Facilitates” and “Facilitates Somewhat” on a five-point scale of 1 “Greatly Inhibits to 
5 “Greatly Facilitates.” 



* Includes those indicating “Greatly Inhibits” and “Inhibits Somewhat” on a five-point scale of 1 “Greatly Inhibits to 5 
“Greatly Facilitates.” 



In interviews, principals and district supervisors highlighted the role of the state in the 
development of their science programs through a variety of mechanisms: state standards; the 
Pennsylvania System of School Assessment (PSSA); and funding the SIE program. As several 
noted: 



Definitely receiving the state grant [SIE] has helped... we would not have been able to 
afford this otherwise. [Principal] 

Probably one of the main things that has helped is the state actually having the PSSA 
with science counting now. . .It makes it a lot easier to say, “We don ’t have a choice, this 
counts. ” [District Supervisor] 

Really, it has been driven by the state standards... with the grade Pennsylvania 
assessments coming on line, that also kind of gave a little bit of urgency here to get some 
things moving; it has mostly been driven by state requirements. [District Supervisor] 

Although the introduction of the science PSSA has increased the emphasis science instruction 
receives, principals and district supervisors also indicated that because the science PSSA does 
not count toward Academic Yearly Progress, instructional time for science in their schools was 
still limited. As two stated: 

The bottom line is AYP, we need to show growth. So, our focus is the language arts area 
and math and then comes science and then comes social studies. [Principal] 
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The biggest challenge is really we have to focus so much on the reading and the math just 
because that is what we are held accountable for... I guess that is our biggest challenge, 
seeing how science fits in with it and convincing... every one that science is just as 
important and that it will help in reading and math, but we are just so narrowed right 
now. [District Supervisor] 

Principals were also asked what additional services the SIE program could provide that would 
help them support their school’s science program. Ideas and tools for securing resources for 
science instruction; assessing progress of the school’s science program; enhancing teacher 
collaboration; and tools for providing feedback to teachers were each mentioned by roughly 60 
percent of principals. (See Table 4.) (Principals’ responses to this questionnaire item 
disaggregated by cohort can be found in Appendix A.) 



Table 4 



Principals Indicating that Potential Program Services Would be Very Valuable 





Percent of Responding 
Principals 
(N = 100) 


Ideas and tools for securing the resources needed for science instruction 


59 


Ideas and tools for assessing progress of our school’s science program 


59 


Ideas and tools for creating opportunities for teacher collahoration 


59 


Ideas and tools for observing and providing feedback to teachers about their science 




instruction 


59 


Professional development for me to increase my own understanding of how to integrate 




science with mathematics and literacy 


55 


Ideas and tools for increasing parental support/interest/involvement in our school’s science 




program 


44 


Professional development for me to increase my own understanding of effective science 




teaching 


44 


Ideas and tools for leveraging other district resources (e.g., professional development days) 




to support science instruction 


44 


Ideas and tools for increasing community support/interest/involvement in our school’s 




science program 


42 


Ideas and tools for making science more of a priority in the school 


36 


Professional development for me to increase my own science content knowledge 


24 



Professional development for principals was also seen as important. Fifty-five percent of 
responding principals indicated that training to increase their understanding of how to integrate 
science with other subjects would be of value; and 44 percent indicated that professional 
development to increase their understanding of effective science teaching would be useful. This 
need was also identified by many district science supervisors. As one supervisor stated: 

We plan on sending our principals to some professional development as well, because I 
think they need to understand the whole approach and where we are headed. . . The 
Foundations piece in particular. . . That is the piece we think is so very important for the 
principal, because if they are going to go in and in any way effectively evaluate a 
teacher’s implementation of this, they need to experience that piece of the professional 
development. 
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The Quality of SIE Professional Development 



The main goal of the SIE professional development program is to move teachers from 
mechanical to purposeful implementation of science modules. Many teachers achieve the 
mechanical use of science modules by following the steps in the teacher’s manual and having 
students complete the activities. Purposeful use involves going beyond just doing the activities 
to having students make sense of the ideas targeted in a given activity and how those ideas build 
toward bigger scientific principles. Students should also reflect on how their understanding of 
the targeted science concepts has changed given their experiences with the module, and be 
helped to make connections across module activities in the development of science ideas. 

Purposeful implementation of a module requires considerable knowledge and skill on the part of 
teachers. First, teachers need to know and feel comfortable with the mechanics of a module — 
’’what’s in the box,” i.e., how to manage materials and students while using a module. 

Second, before they can effectively teach the science content to students, teachers must 
themselves understand the content. At a minimum, purposeful use of a module requires that 
teachers understand the ideas students are expected to learn, and how those ideas fit together in a 
coherent and cohesive framework. Teachers would also benefit from understanding the content 
beyond the student level, including how the ideas progress through the K-12 sequence, so they 
can guide students along productive paths of inquiry. 

Third, in order to teach effectively, teachers need to have a clear understanding of how students 
learn science and an explicit vision of effective instruction. A great deal has been learned in the 
last few years about how people learn and the implications of research in the cognitive sciences 
for instruction. Key findings that have emerged from this research indicate that effective 
instruction provides motivation to learn, elicits students’ prior knowledge, intellectually engages 
students with important science ideas, emphasizes the use of data as evidence to support/critique 
conclusions, and provides opportunity for students to make sense of the targeted ideas. ^ 

For instruction to be purposeful and effective, teachers need to be able to integrate their 
knowledge of the science content and their familiarity with the nuts and bolts of the module. As 
a result, teachers would understand the “content storyline,” how each of the scientific ideas 
addressed by the module is developed through specific learning experiences. Ideally, they would 
be able to trace the development of a scientific idea through a module (e.g., students’ initial ideas 
about a particular idea are elicited in activity 1; the idea is then developed in activity 2; students 
have an opportunity to practice and master the idea in activity 3). 

Similarly, for instruction to be purposeful and effective, teachers need to be able to integrate 
their knowledge of the science content with their understanding of how students learn science. 



* National Research Council. (2000). How people learn: Brain, mind, experience, and school. J. D. Bransford, A. L. 
Brown, & R. R. Cocking (Eds.). Washington, DC: National Academy Press. 

National Research Council. (2005). How students learn: Science in the classroom. M. S. Donovan & J. D. 
Bransford, (Eds.) Washington, DC: National Academy Press. 
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This type of knowledge is commonly referred to as pedagogical content knowledge; it includes 
recognizing initial ideas students are likely to have that may get in the way of their learning the 
targeted concepts (often termed “misconceptions” or “naive ideas”), and specific strategies for 
helping students develop the correct ideas. For example, students often think that connecting a 
light bulb to one end of a battery will cause the bulb to light. A specific strategy for helping 
students move past this idea is to have them try many different configurations of connecting the 
bulb to the battery, recording which do and do not work, and then carefully analyzing the results 
to find the commonalities in the configurations that work, and the commonalities in those that do 
not work. 

Finally, it is important to note that general pedagogical skills are necessary for effective teaching 
of any topic. The SIE program addresses two specific sets of general pedagogical skills: the 
integration of reading/language arts instruction with the teaching of science (e.g., notebooking 
strategies) and assessment strategies. Figure 2 shows a progression of how these elements build 
toward purposeful implementation of a module. 



Knowledge and Skills Needed for Purposeful Implementation of a Module 
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Figure 2 
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The SIE program is continually reflecting on its work and making changes to ensure that the 
professional development is as effective as possible in helping teachers reach the goal of 
purposeful use of the modules. As part of this process, the program has redesigned the 
Foundations course to focus more explicitly on helping teachers develop a common vision of 
effective science instruction. HRI observed the revised Foundations course during the 2009 
ASSET Feadership Conference, which is described in the next section of this report. This 
section of the report focuses on data about the Initial Module Trainings and Module Enrichment 
Trainings. 

During Year Three of the program. Cohort 3 teachers attended Initial Module Training for their 
first module and Cohort 2 teachers attended Initial Module Training for their second module. 
Cohort 1 teachers, most of whom had taught their first SIE-provided module two times, had the 
option to attend Module Enrichment Training for that module. A post-professional development 
questionnaire was administered to everyone attending one of these workshops.^ Table 5 shows 
the number of teachers completing the questionnaire at each type of workshop. 



Table 5 



Responses to Post-Professional Development Questionnaire 





Number of Responding Teachers 


Initial Module Training on R' module 


585 


Initial Module Training on 2'“’ module 


964 


Module Enrichment Training on E' module 


83 



Initial Module Trainings 

The primary purpose of the Initial Module Trainings is to prepare teachers to use the 
instructional materials for the first time, including familiarizing them with the activities, the 
targeted science content, and strategies for engaging students with the content. Data on the 
quality and impact of the Initial Module Trainings come from questionnaires completed by 
teachers attending these trainings, interviews with samples of teachers and principals, and HRTs 
observations of a sample of workshops. 

Teacher Expectations 

Understanding the expectations of teachers participating in SIE may help the program better plan 
to address their needs. Cohort 3 teachers were asked on an open-ended item on the baseline 
questionnaire to explain what they hoped to gain from participating in SIE professional 
development. The most common responses, each given by approximately one-third of the 
respondents, were that they hoped to learn how to use their module and gain new instructional 
strategies for teaching science. Typical responses included: 



^ Because the post-professional development questionnaire was administered to a captive audience, the response rate 
should be close to 100 percent. 
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A deeper understanding of content/procedures. Knowledge of the kit that will allow me 
to organize/implement more efficiently. 



1 wanted to see the materials, how to implement them. 1 wanted to see how the notebook 
was utilized, and different things that I could do to make the program successful in my 
classroom. 

I’m hoping to gain a better understanding of how to facilitate my students’ learning and 
organize the modules so our time spent in science can be maximized. 

One-fifth of the questionnaire respondents indicated that they hoped to learn how to make 
science more enjoyable and engaging for their students. As two participants wrote: 

A better understanding of how to bring all of this information back to my students and 
understand how to keep their interest in it!! 

Knowledge and materials that will allow me to excite students and get them to want to 
learn science. 

Teacher and Principal Perceptions of the Quality of the Initial Module Trainings 
Overall, teachers and principals had very positive opinions about the SIE Initial Module 
Trainings. Teacher responses to a questionnaire administered at the end of each training 
highlighted the quality of the facilitators. (See Table 6.) Nearly all respondents thought 
facilitators provided useful tips for successfully implementing the modules, encouraged active 
participation, and adequately addressed questions about the science content in the modules. In 
addition, the vast majority of teachers indicated that the Initial Module Training sessions 
increased their familiarity with the activities in the modules and prepared them to use the 
teaching strategies promoted by SIE, indicating teachers’ expectations for the professional 
development had been met. Teachers also thought the workshops had increased their 
understanding of the science content in the modules. (Post-professional development 
questionnaire data on the Initial Module Trainings are reported by session type in Appendix A.) 
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Table 6 

Teacher Opinions of the Quality of SIE Initial Module Training 





Percent of 
Responding 
Teachers Agreeing^ 
(N = 1,545) 


The facilitators shared tips and suggestions for successfully implementing the module. 


97 


The goals of this workshop were clear. 


96 


This workshop familiarized me with the activities and materials in the module. 


96 


The facilitators encouraged active participation and investigation by all participants. 


95 


This workshop increased my understanding of the science content in the module. 


95 


The various components of this workshop were useful in meeting its goals. 


94 


The facilitators were well prepared. 


94 


Question about the science content in the module were adequately addressed by the facilitators. 


94 


This workshop prepared me to use the teaching strategies promoted by SIE. 


94 


This workshop was relevant to my classroom instruction. 


94 


This workshop increased my confidence in my ability to teach using the SIE module. 


93 


This workshop was worth the time that I invested. 


92 


I would recommend this professional development to a colleague. 


92 


This workshop reflected careful planning and organization. 


92 


The facilitators modeled effective teaching strategies. 


92 


The facilitators explicitly discussed how, when and why to use different teaching strategies. 


91 


There were opportunities for participants to express their views and collaborate with peers. 


89 


Adequate, time, structure, and guidance were provided for participants to discuss the SIE modules 




and pedagogical strategies with each other. 


81 


Adequate time, structure, and guidance were provided for participants to reflect individually on 




the substance of this workshop. 


80 



Includes those indicating “Agree” and “Strongly Agree” on a five-point scale of 1 “Strongly Disagree” to 5 “Strongly Agree.” 



In addition, approximately two-thirds of respondents indicated that the pace of the professional 
development was appropriate. (See Table 7.) Seventeen percent of participating teachers found 
the pace of the Initial Module Trainings too slow; 19 percent indicated the pace was too fast. 



Table 7 

Teacher Opinions of the Pace of SIE Initial Module Trainings 





Percent of Responding Teachers 
(N = 1,494) 


Much too slow 


2 


Somewhat too slow 


15 


Appropriate 


64 


Somewhat too fast 


14 


Much too fast 


5 



When asked on the post-professional development questionnaire and in interviews what aspects 
of the workshops were most effective in preparing them to implement their module, teachers’ 
most common responses were going through the module activities as if they were students, and 
having experienced classroom teachers facilitate the sessions as they provided many practical 
implementation tips. In the words of three teachers: 
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It was most helpful to actually perform each lesson as the student would and discuss 
ways in which students would respond. It was also helpful to get comfortable with the 
materials being used. 

They did a nice job telling us how to set up each lesson and the notebooks. How 
important that was, and they did a nice job of getting us excited about it. They did a nice 
job walking us through the experiments, emphasizing the points we needed to make. I 
liked how they helped us set up our notebooks. The notebook was key; we referred to it a 
lot to help us through each lesson. 

All the presenters we had were very good, probably because they are in the classroom 
currently. It’s someone who actually does the module. 

SIE encourages principals to attend the professional development provided to teachers, and 42 
percent of respondents to the Principal Questionnaire indicated they had attended an Initial 
Module Training during the 2008-09 academic year (in 2007-08, 40 percent of responding 
principals indicated they had attended one). Nearly all of the interviewed principals who had 
attended an Initial Module Training noted that they thought the professional development was 
very effective at preparing teachers to implement their modules. As one principal said: 

It was probably the best professional development workshop that they [teachers] have 
ever attended because it actually got them ready for what is going to happen in the 
classroom. 

Further, many indicated that the sessions increased their own understanding of the modules and 
what teachers would be doing in the classroom; a few also mentioned that they felt better 
equipped to help teachers in the classroom. As three principals commented: 

7 think it is important for administrators to be in the training to see what the teachers are 
trying to do and what sort of supports and needs they would have. I can ’t offer support if 
I don ’t understand the program. 

Well, [attending the training] helped put me on the same playing field as the teachers, so 
I know what the kits are supposed to look like when they are implemented... [it] provided 
some common language for us. 

[It] helped me to know what I should then be looking for when I am in a classroom and 
science is being taught. 

When asked how the Initial Module Trainings could be more effective, most teachers indicated 
that they were satisfied and did not see a need for changes. Still, a number of teachers did offer 
ideas for improvement. The most common suggestion, provided by 25 percent of survey 
respondents, was to spend more time going through the module activities. This suggestion came 
from teachers attending both one- and two-day workshops, with many indicating that the 
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trainings did not use time effectively, often rushing through lessons. The following responses 
were typical of these teachers: 

Maybe extend the time of the training. It seemed the presenters are crunched into a 
certain number of days and they had to rush through each lesson, and it was done so 
quickly that when you do it with your class you have some trouble remembering 
everything. I say, extend the time to not be so crunched. 

Trying to get all the info in on one day was hard... They started off doing a nice job 
walking us through it, but by the end they moved too quickly. They would do an 
experiment then move onto the next one and we didn ’t get to write down notes. 

Addressing Teachers’ Concerns about Implementing the SIE Program 
As part of the baseline and end-of-year questionnaires, HRI administered a set of items 
developed as part of the Concerns Based Adoption Model (CBAM). The theory holds that when 
teachers adopt a new curriculum or instructional approach, they progress through a series of 
concerns about the “innovation.” As experience with the innovation grows, early concerns (e.g., 
being unfamiliar with the curriculum) are resolved, and later ones emerge. The implication of 
this stage theory is that early concerns must be resolved before later ones can be effectively 
addressed. The CBAM theory suggests that effective professional development specifically 
targets the concerns that are likely to emerge at different points in the adoption process. The 
seven stages of concern are operationalized in a 35-item CBAM questionnaire developed to 
measure the intensity of each concern: 

1. Awareness (e.g., I don’t even know what the SIE modules are.) 

2. Informational (e.g., I would like to know what resources are available for implementing 
the SIE modules.) 

3. Personal (e.g., I would like to know how my teaching is supposed to change using the 
SIE modules.) 

4. Management (e.g., I am concerned about my inability to manage all that the SIE modules 
require.) 

5. Consequence (e.g., I am concerned about how the SIE modules affect students.) 

6. Collaboration (e.g., I would like to develop a working relationship with both our faculty 
and outside faculty using SIE modules.) 

7. Refocusing (e.g., I would like to modify our use of the SIE modules based on the 
experiences of our students.) 

The hypothetical concerns profile in Eigure 3 illustrates how early concerns (e.g., informational) 
are the most intense early in an implementation, but decrease over time, and are replaced by 
other concerns. It is important to note that in the CBAM theory, the changes depicted in Eigure 3 
can take two to three years to be observed. 



*** HRI modified the original CBAM questionnaire to tailor it to the SIE program. 
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Hypothetical Concerns Profile at 




Type of Concern 



-♦ — Early — ■ — Middle 



Figure 3 



Figure 4 shows CBAM data from Cohort 3 teachers. Based on these data, SIE was successful at 
addressing these teachers’ initial concerns about the program and the implications for classroom 
practice. This change in the concerns profile is similar to what was observed for Cohorts 1 and 2 
teachers during their first year of participation in the program. 



Cohort 3 




Types of Concern 

— ■ — 2008-09 Baseline — ▲ — 2008-09 End-of-Year 

2008-09 End-of-Year profile is significantly different than 2008-09 Baseline profile, p < 0.05 
(MANOVA) 

Figure 4 
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Observers’ Perceptions of the Quality of the Initial Module Trainings 

The SIE Initial Module Trainings are focused largely on familiarizing teachers with the basics of 
the module. The primary goals for the Initial Module Trainings are to: 



1. Prepare teachers to implement the module at the mechanical level; 

2. Deepen teachers’ understanding of the targeted science content in the module; and 

3. Prepare teachers to use instructional strategies that support module implementation. 

Evaluation staff observed portions of 28 Initial Module Training sessions between December 
2008 and Eebruary 2009. HRI attempted to observe Initial Module Trainings for as many 
different modules as possible. However, a combination of the program providing more sessions 
for some modules than others, and the fact that HRI tried to maximize the number of workshops 
observed in each trip, resulted in Initial Module Trainings for certain modules being observed 
more often than others. Table 8 summarizes the number of sessions observed for each module. 



Table 8 



Initial Module Trainings Observed 





Number of Sessions 


Chemical Tests 


2 


Changes 


2 


Ecosystems 


1 


Electric Circuits 


4 


Human Body 


1 


Levers and Pulleys 


2 


Mixtures and Solutions 


2 


Motion and Design 


2 


Plant Growth and Development 


1 


Rocks and Minerals 


4 


Solids and Liquids 


2 


Variables 


2 


Water 


2 


Weather 


1 



HRI utilized a framework based on research on how people learn to examine the extent to which 
the SIE professional development achieved its goals. This framework asserts that effective 
science professional development must provide sufficient opportunities for teachers to engage 
intellectually with important and accurate scientific and pedagogical ideas; reflect on and make 
sense of the key ideas of the session; and relate the ideas in the session to their work as 
classroom teachers. An analysis of SIE Initial Module Training professional development for 
each of its goals follows. 



** As described previously in this report, HRI’s frame for thinking about the quality of instruction for both adult and 
student learners is based upon the National Research Council’s How people learn: Brain, mind, experience, and 
school (1999) and How students learn: Science in the classroom (2005). 
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^ Preparing teachers to implement the module at the mechanical level 
Familiarizing teachers with the activities and materials in the module is an essential step in 
preparing teachers to implement a module. Based on HRI’s observations, the trainings appeared 
to be very successful at orienting teachers to the materials in the modules they would be using, as 
well as the mechanics of implementing the various investigations in their classrooms. 

The sessions typically started by having the participants walk through the various sections of the 
teacher’s manual, gaining an overview of what was addressed in the module. The teachers then 
experienced most of the student activities first hand. This design allowed participants to become 
familiar with the set up and use of materials, as well as the structure and flow of the lessons. 

Tips for facilitating the use of the module with students were also provided by the facilitators, 
and in some cases by other participants who had experience with the module. These tips 
included suggestions for preparing the materials needed to conduct the investigations, efficiently 
distributing materials to students, and simplifying journaling for beginning writers in early 
primary grades. 

Although the Initial Module Trainings provided a great deal of opportunity for teachers to learn 
how to set up and conduct the activities in their classrooms, there was less opportunity for them 
to learn how to debrief the findings from activities and help students draw appropriate 
conclusions. When sessions did focus on this aspect, the quality was typically good. In these 
sessions, facilitators modeled and then explicitly discussed with participants how to debrief 
module activities with students. For example, in an Ecosystems session, participants shared their 
thoughts on what constitutes an ecosystem after completing a student activity. The facilitator 
modeled how to take the various ideas and put them together to generate a definition for an 
ecosystem. The facilitator then explicitly pointed out that this collaborative approach for 
developing vocabulary was a good strategy to use in their classrooms to help students make 
meaning from their experiences and not just memorize the definition. 

Similarly, in a training on the Mixtures and Solutions module, the facilitators explicitly discussed 
how to help students make sense of conflicting data. As the observer described: 

Participants were provided a “mystery chemical” (Epsom salts) to identify, and a data sheet showing 
various properties of six different substances. Groups wrote out a plan to identify the chemical, 
implemented their plan, and then reported their results to the large group. The groups noticed a great deal 
of variation in their results; for example the mass of the mystery chemical needed to saturate 50mL of 
water ranged from 34 to 46 grams. The facilitators helped participants consider how they could carry out a 
discussion in their own classrooms when students reported conflicting results. In particular, they discussed 
appropriate questions to ask students such as, “What might have affected the results?” and the importance 
of placing some responsibility on the students for searching for possible answers, instead of providing 
students with the “correct” results. 

However, it was more typical in the observed sessions for facilitators, after having participants 
conduct a hands-on activity, to tell them to think about how they would wrap up lessons with 
their own students and then immediately move on to the next activity. In these cases, the 
facilitators neither provided an opportunity for the participants to make sense of the science 
content as adult learners nor allowed participants to consider how they could help students make 
sense of the ideas. The facilitators often said things like, “Students should now reflect and write 
a conclusion in their notebooks,” or “Now you would have them [students] look at the focus 
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question and write an answer to it,” but did not model or discuss how to carry out the 
suggestions. Facilitators appeared to understand the importance of reflection and sense-making, 
as was evidenced by their frequent reference to them during the trainings, but there was rarely 
adequate time provided for these components. For example: 

During a two-day training on the Chemical Tests module, participants engaged with learning how to 
implement the module activities by working through them as students. At the conclusion of each student 
activity, the facilitator provided implementation suggestions, such as labeling spoons to match the labels on 
the bottles of unknown solids to prevent cross-contamination and conducting the lesson on crystals on a 
Friday to allow evaporation to occur over a weekend. During these discussions, the facilitator also 
encouraged participants to reflect on their understanding of how to implement the materials and seek 
clarification, if needed, on any areas of confusion. Comments from the facilitator included, “I want you to 
reflect as a teacher. . .How would you run this in your classroom? What types of questions could you ask to 
guide the discussion?” and “Step back and think as if you were reviewing this with your students. What 
could you do to get them to reflect on the key ideas related to suspensions and solutions?” Although these 
prompts from the facilitator would likely lead to useful thoughts on the part of the teachers, the participants 
were not provided with workshop time to reflect and respond to these prompts. Instead, the facilitator 
moved onto the next student lesson within a minute or two of posing these questions; teachers were neither 
given the time to reflect individually nor asked to share their ideas with the group at large. 

Overall, familiarizing teachers with the activities in the module, and providing them with 
practical teacher tips, were strengths of the Initial Module Training sessions. The program 
should consider ways to ensure that all sessions provide both the structure and the time for 
teachers to reflect on how to implement all aspects of the activities, including the sense- 
making/wrap up portions of the lessons. 

> Deepening teachers’ understanding of the targeted science content in the module 
Purposeful implementation requires teachers to have a solid understanding of the science content 
addressed by the module. In order to achieve this understanding, just like students, teachers must 
be intellectually engaged with the targeted science content and have opportunities to make sense 
of the content. 

The Initial Module Trainings used a number of strategies to engage teachers with the science 
ideas in the module. One successful strategy was having teachers engage with the targeted 
science content by completing the module activities as if they were students. Another good 
strategy was the use of focus questions at the beginning of activities provide participants with a 
purpose for the lessons. These focus questions helped to ensure that teachers were intellectually 
engaged with the targeted content as well as the mechanics of the module activities they 
completed. For example, the focus questions for an investigation in the Water module were: (1) 
“Does water always flow downhill?” and (2) “How does changing the slope or quantity of water 
change the speed at which it flows downhill?” These questions were directly linked to the 
activity involving manipulating the angle of the slope of a ramp on which water was dropped, as 
well as the amount of water running down the slope. 

However, in several sessions, the facilitators short-circuited participants’ intellectual engagement 
with the targeted science ideas by telling them what should be learned from the activity before 
they had the opportunity to analyze their data and think about the science ideas in the activity. 

For example: 
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The participants in an Initial Module Training on the Electric Circuits module completed a lesson in which 
they tried to light a bulb with a battery and wire. The targeted concept to be learned from this lesson was 
that a complete circuit includes an energy source and allows electric current to flow only if it has a 
complete loop through which it can pass. As the participants were collecting data on which arrangements 
of the materials would and would not light the bulb, the facilitator told the participants, “The wire has to be 
touching the bulb on the bottom and side.” By giving the participants the answer before they finished the 
activity and had an opportunity to draw conclusions from their data, the facilitator rather than the teachers 
did the intellectual work. 

Furthermore, in order for teachers to learn the targeted science ideas, the science content they 
engage with must be accurate. In most of the sessions, the science content teachers were 
engaged with was accurate; however, a few observed trainings included content inaccuracies that 
hindered participants’ ability to learn the science concepts. For example, in a number of 
observed sessions of the Electric Circuits module training, the term “electricity” itself was not 
clearly defined and it was often used interchangeably for “electric current” and “electric energy.” 
Although these terms may appear similar, they represent very different scientific phenomena. In 
another case, the facilitator of a Motion and Design training defined inertia as, “an invisible force 
that keeps your momentum going.” This statement is problematic for a number of reasons. 

First, inertia is not a force. Second, this statement implies that a force is required to maintain 
motion. Both the definition of a force and the effect of a force on motion are ideas that students 
have difficulty learning, and research has shown that students (and adults) have misconceptions 
about these ideas that are very resistant to change. By not being careful with their language, the 
facilitators decreased the likelihood that the teachers would develop the correct scientific 
understanding themselves, much less be able to help students learn those ideas. 

In addition to being intellectually engaged with the targeted science concepts, learners need 
opportunities to make sense of those concepts in order to develop a sound understanding. 
Opportunities for participants to reflect on the meanings of the activities were inconsistent in the 
observed sessions. In some cases, participants were provided structured time to discuss and 
reflect on the science concepts targeted by the module activities. For example, in a Levers and 
Pulleys session the participants applied what they had learned about levers in a new context in 
order to deepen their understanding of the various classes of levers: 

During a module activity, the participants had a group discussion regarding how the human arm can be 
classified as a lever. After the facilitators explained the types of levers the arm can act as, teachers 
provided their own examples of instances where the arm would be classified each way. For example, one 
participant said that the arm would be a Class- 1 lever when throwing a javelin and the explanation for this 
classification agreed with the targeted concept that a class- 1 lever has the fulcrum between the load and the 
effort. Another explained that the arm would be classified as a Class-3 lever when throwing a discus with 
the explanation supporting the idea that a class-3 lever has the effort between the fulcrum and the load. 

Focus questions, used to engage students with the activities, were also often used as reflection 
questions at the end of the activities to help the teachers make meaning of the activities. 
Sometimes the dual use of these questions worked well. For example, the focus questions 
described previously for an investigation in the Water module (i.e., “Does water always flow 
downhill?” and “How does changing the slope or quantity of water change the speed at which it 
flows downhill?”) directly asked about the science ideas to be learned. By revisiting these 
questions at the end of the activity, the participants were helped to make sense of the targeted 
concepts: (1) water flows downhill; (2) larger amounts of water flow more quickly; and (3) 
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increasing the slope over which water flows makes it flow more quickly. 

In other cases, the focus questions related to the procedures of the activity, rather than the 
science idea. For example, in one observed session the focus question for a lesson in the Rocks 
and Minerals module was: “How can we use simple tools to determine how much light passes 
through a mineral?” This question was also used as the reflection question, and participants’ 
responses centered on the mechanics of using a light pen to look through a mineral rather than 
the more important idea of how scientists could use the way light passed through a mineral to 
help identify it. Especially if the SIE program wants to use the same prompts as both focus and 
reflection questions, it will be important to ensure that the prompts be about the targeted science 
ideas, rather than the mechanics of an investigation. Otherwise, the professional development 
facilitators should be provided with separate reflection questions for helping participants make 
sense of the science content. 

The lack of sense making about the science ideas in an activity was fairly common in the 
observed sessions. In many cases, after participants did the activities from the module, the 
facilitators simply moved on to the next activity. Often, the facilitators told teachers that it was 
important that they help students reflect on the science, indicating they were aware of the 
importance of sense-making, but they typically did not take the time to make sure that the 
teachers understood the science. In cases where the facilitators did lead a post-activity 
discussion, the conversation was often centered on the mechanics of implementing the activity in 
the classroom rather than on the science content. Addressing participants’ concerns about 
implementation is a necessary and important component of the training; however it needs to be 
done in addition to, not instead of, helping teachers make sense of the science. 

> Preparing teachers to use instructional strategies that support module implementation 
Another goal of the Initial Module Trainings was to prepare teachers to use specific pedagogical 
strategies to implement the modules effectively. Understanding what strategies to use, as well as 
when and why to use each of those strategies, will help teachers implement the modules in a 
purposeful manner. 

Overall, the trainings appear to have provided teachers with many opportunities to learn the 
instructional strategies that will help them implement the module. Eor example, every observed 
session included explicit discussions on how to set up and use science notebooks, an important 
way to integrate literacy skills into science teaching. Participants both kept their own science 
notebooks and participated in discussions on using various notebooking strategies with their 
students. In some sessions, facilitators shared questions teachers could ask students to probe 
their understanding, for example, “How did you get that result?” and “Why do you think that is 
true?” 

Although many strategies were shared, observers rarely saw discussions of when a strategy 
should be used, or why a particular strategy would be appropriate in a given situation. Eor 
example, while “power conclusions” were discussed in several observed sessions, it was not 
made clear to participants when a power conclusion should be used, i.e., at the end of a series of 
lessons, not at the conclusion of the intermediate lessons. Eor more purposeful use of 
instructional strategies, SIE should introduce strategies on a “need to know” or “just in time” 
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basis and be explicit about the purpose of the strategy (e.g., this strategy is a way to help students 
make sense of the content, this strategy is useful in eliciting students’ initial ideas). 



Overall, the Initial Module Trainings appeared to be providing the participants with ample 
experiences to engage them with the set up and implementation of the module activities, the 
content within the modules, and instructional strategies to facilitate science instruction. The 
elements of effective professional development were present in sessions; however the time 
dedicated to and quality of the reflection on what science is being taught through the module 
activities and how to best help students understand that science were inconsistent. Participants in 
SIE professional development would benefit from the facilitators incorporating explicit reflection 
on the teaching and learning of the targeted science ideas into the sessions. 

One effective strategy to structure reflection for participants, which has been used in some SIE 
sessions, is the “learner hat/teacher hat” approach. This strategy consists of teachers first 
engaging in a module activity from the student perspective, including providing opportunities to 
make sense of the science content, and then discussing the design and facilitation of the activity 
from the teacher perspective. The use of this method creates a structure to, and expectations for, 
professional development that increases the likelihood that all goals will be addressed 
adequately, and that the science content will not get lost. Although some SIE sessions have 
employed this strategy to some degree, the program would benefit from the use of this strategy in 
a more formal and consistent manner. Eollowing is an example of a Levers and Pulley training 
session where this strategy was used: 

Before engaging participants with any of the module activities one facilitator made the follow remarks 
regarding the two perspectives that participants would be taking during the training: 

Facilitator: “You are going to be wearing two different hats today, teacher hat and adult learner, not a 5* or 
6* grade learner. [Later when starting with the first investigation] Let’s close our manual and put on our 
adult learner hat.” 

Throughout the two-day session, participants would complete student activities as learners focusing on 
understanding the science content, then talk about teaching tips, and on occasion instructional strategies or 
what concepts were being covered, from the teacher’s point of view. The facilitators were explicit about 
when the participants were switching between their role as learner and as teacher. For example, one 
facilitator addressed the group by saying, “We are going to go through some power point slides with some 
content. . .It is always good when the teacher knows more about what they are teaching than the 
students. . .This is not something you would share with you students. . .Once again this is knowledge for 
you, not your students.” 

Using the learner hat/teacher hat approach has implications for time allotments during a session. 
Using this approach requires facilitators to go slower through individual module activities, 
spending time both modeling lessons (learner hat) and debriefing lessons (teacher hat). 
Consequently, facilitators may not be able to cover all lessons in a module using this approach in 
a two-day session. 

Given that not all lessons may be covered during the two-days, the trainings might be structured 
so that this approach is used only for selected lessons, i.e., those that cover science ideas that are 
particularly challenging and/or important, and those that are the most difficult to implement from 
a logistics standpoint. Eor other lessons, facilitators would go faster, perhaps simply describing 
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the lesson briefly or asking participants to review it on their own (e.g., in the evening after the 
first day). If SIE decides to take this approach, it will be important that a plan for the Initial 
Module Training for each module be developed that makes optimal use of the available time. 

Lastly, in some sessions, facilitators dealt with science content beyond the student level, though 
they did not always make a clear distinction between what content was meant for students and 
what content was meant only for the participants. Moreover, in a few instances, teachers were 
encouraged to share age-inappropriate content information with students. For example, a 
facilitator leading a Water module training used a PowerPoint presentation to describe the 
molecular properties of water. The facilitator noted that the module did not teach the molecular 
structure of water and suggested that the teachers use supplemental activities and PowerPoint 
slides to illustrate the structure and arrangement of water molecules in each different phase of 
matter. However, an understanding of molecules and their structure is not part of the 4* grade 
standards in Pennsylvania, and is likely developmentally inappropriate for students who would 
be receiving instruction with this module. It will be important for the program to make sure that 
all the facilitators point out to participants what content information is for their own benefit and 
what is appropriate for students. 



Module Enrichment Trainings 

Cohort 1 teachers were offered a two-day Module Enrichment Training on their initial module. 
The Module Enrichment workshops are intended to move teachers further along the continuum 
from mechanical to purposeful implementation of the SIE-supported modules by deepening 
teachers’ understanding of the science content, the content storyline, and pedagogical content 
knowledge related to their module. The Module Enrichment workshops were offered for the first 
time during the 2007-08 school year, and about half of Cohort 1 teachers eligible to attend a 
Module Enrichment Training did so. The remaining Cohort 1 teachers were invited to attend the 
Module Enrichment Trainings this year; HRI received completed post-professional development 
questionnaires from 83 participants. 

This section of the report describes the quality of the SIE Module Enrichment professional 
development using data from teacher questionnaires and teacher interviews. HRI did not observe 
any Module Enrichment sessions this year given the timing of the sessions and the program’s 
decision to focus observation efforts on the Initial Module Trainings. 

Teachers’ Needs and Expectations 

Six out of the 10 interviewed Cohort 1 teachers had attended a Module Enrichment Training. 
When asked what they had hoped to get out of the training, these teachers had varying 
expectations. Several said they had hoped to have time to debrief and share their own 
experiences with the module; others did not know what to expect; and one wanted 
implementation tips to improve their use of the module. 

The fact that teachers did not have a clear understanding of the purpose of the Module 
Enrichment Trainings may explain the relatively low attendance at these sessions, as teachers are 
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not aware of what the value added of attending would be. The program may want to consider 
ways to clarify and communicate the goals of the Module Enrichment Trainings. 



Teachers’ Perceptions of the Quality of the Module Enrichment Trainings 
A majority of participating teachers expressed positive opinions about the Module Enrichment 
Trainings on a post-professional development questionnaire administered at the end of each 
training. Responses were similar to those from teachers who had attended the Initial Module 
Trainings. Eor example, nearly all thought that facilitators were well prepared, encouraged 
active participation, shared tips for implementing the module, and adequately addressed 
questions about the content in the modules. (See Table 9.) 



Table 9 



Teacher Opinions of the Quality 
of SIE Module Enrichment Professional Development 





Percent of 
Responding 
Teachers Agreeing^ 
(N = 83) 


The facilitators were well prepared. 


99 


The facilitators encouraged active participation and investigation by all participants. 


98 


The facilitators shared tips and suggestions for successfully implementing the module. 


98 


Question about the science content in the module were adequately addressed by the facilitators. 


98 


This workshop reflected careful planning and organization. 


98 


The goals of this workshop were clear. 


96 


There were opportunities for participants to express their views and collaborate with peers. 


96 


Adequate, time, structure, and guidance were provided for participants to discuss the SIE 




modules and pedagogical strategies with each other. 


95 


Adequate time, structure, and guidance were provided for participants to reflect individually on 




the substance of this workshop. 


95 


The facilitators explicitly discussed how, when and why to use different teaching strategies. 


95 


The facilitators modeled effective teaching strategies. 


95 


This workshop was relevant to my classroom instruction. 


95 


The various components of this workshop were useful in meeting its goals. 


94 


This workshop increased my understanding of the science content in the module. 


93 


This workshop prepared me to use the teaching strategies promoted by SIE. 


93 


This workshop increased my confidence in my ability to teach using the SIE module. 


89 


This workshop familiarized me with the activities and materials in the module. 


87 


I would recommend this professional development to a colleague. 


85 


This workshop was worth the time that I invested. 


84 



Includes those indicating “Agree” and “Strongly Agree” on a five-point scale of 1 “Strongly Disagree” to 5 “Strongly Agree.” 



A majority of teachers found the pace of the sessions to be appropriate; although 1 in 5 thought 
the pace of the workshop was too slow. (See Table 10.) 
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Table 10 



Teacher Opinions of the Pace of SIE Module Enrichment Trainings 





Percent of Responding Teachers 
(N = 83) 


Much too slow 


2 


Somewhat too slow 


18 


Appropriate 


75 


Somewhat too fast 


1 


Much too fast 


4 



Teachers were asked which aspects of the training were most effective. The most frequent 
response to this open-ended question, given by 38 percent of respondents, was the opportunity to 
share their experiences and ideas for implementing the module. One-quarter of the respondents 
noted that the discussions of the big ideas and concepts students should learn in the modules 
were very helpful. Other fairly common responses highlighted the quality of the facilitation, the 
deepening of their content knowledge, and the instructional strategies learned. 

Interviews with participants illustrate these findings. In the words of four attendees: 

The instructors gave me many useful ideas and tips which will enable me to do a much 
better job teaching [the module]. My use of the notebook was also enhanced. This 
workshop was very valuable in explaining [content of module] . 

What we really, really liked about the enrichment was being able to talk to other teachers 
in other schools and finding out the ways they did the modules that worked. We got more 
ideas. We shared ideas. 

Finding out (determining) the big ideas and learning goals that I need to be sure my 
students understand is a very valuable bit of information! Focus questions (shared from 
a peer) were very helpful. The presenters also had us use several strategies that we can 
now use with our classes. 

I really liked the clarification of the “big ideas” in the module. It will also be helpful to 
have a better understanding of exactly what the students are to get from each lesson. I 
have to admit that I have skipped lessons before. Now, I realize the importance of them 
all and how they relate to each other. 

Participants were also asked to reflect on aspects of the workshop that they thought could be 
improved upon. More than half of the respondents (54 percent) indicated that they did not think 
the workshop needed any improvement. Of the suggestions for improvement that were offered, 
the most common, each given by 21 percent of the respondents, were to spend more time sharing 
implementation tips and debriefing the use of the modules. For example: 

More time with peers to share tools, tips, materials that enrich the module, i.e. encourage 
participants to bring samples and enrichment materials. 
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Build time into the workshop for us to talk about what we did with the modules and share 
how to solve situations, problems that come up. Some others might have come up with a 
good way to handle things like how to get all the materials to 30 kids and do the 
experiments in a short time. I wanted to hear more from others. 

Teachers also suggested, both on the questionnaire and in interviews, that the Module 
Enrichment sessions could eliminate activities that were redundant with the Initial Module 
Training. In the words of two teachers: 

This workshop was just a review. It definitely needs to be more challenging. We need to 
share ideas with each other, not review. This is enrichment not the first workshop! 

We basically did the same thing over again. . .If that second one would be not so much of 
a repetition, but add more to what you are learning, it would be better. 

Cohort I and 2 teachers were asked in interviews if they planned on attending a Module 
Enrichment Training the next time they are offered, and, if so, what they hoped to gain from that 
training. Most indicated they would attend when these trainings were next offered, indicating 
that they hoped to learn implementation tips on how to better use the modules. Of those teachers 
indicating they would not attend, travel logistics was the primary reason. 
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The asset Leadership Conference 



In an effort to accommodate the growing number of participants in the SIE program, and to 
bolster school-level support for the program and elementary science education improvement 
more broadly, SIE is developing a cadre of teacher leaders. The program aims to develop these 
teachers for two purposes: (1) to serve as facilitators of SIE Initial Module Training; and (2) to 
be advocates for science education reform in their local settings. The main venue for 
accomplishing these goals is the ASSET Eeadership Conferences (AEC). This section of the 
report is based on HRTs observations of the three-day AEC held in April 2009; it focuses on the 
quality of the teacher leaders’ professional development experience for each purpose. 



Development of Teacher Leaders: Facilitators of Initial Module Training 

ASSET values teachers teaching teachers, and as SIE continues to expand to more schools across 
the state, the plan is to develop a subset of SIE teachers as facilitators of Initial Module Training. 
These teacher leaders have participated in SIE-provided professional development and they have 
implemented at least one module in their own classroom. The teacher leaders have also observed 
Initial Module Trainings from the leader perspective, and in some cases have already helped co- 
facilitate sessions with ASSET staff. 

In thinking about the development of teachers as facilitators of Initial Module Training, it is 
helpful to consider what knowledge and skills are needed for this task. Eigure 5 shows what HRI 
believes teacher leaders need to know and be able to do to be successful in this role. The 
essential components needed for purposeful facilitation of an Initial Module Training are the 
same as those required for purposeful implementation of a module, with the focus being the 
content and activities of a workshop rather than a module. 
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Knowledge and Skills Needed for Purposeful Facilitation of PD for a Module 




Figure 5 



At the 2009 ALC, ASSET devoted a day to piloting the revised Foundations course. The new 
Foundations course focuses on the elements of effective science instruction: motivation, 

eliciting students’ prior knowledge, intellectual engagement, use of evidence to critique claims, 
and sense-making. Having the teacher leaders experience this revised course helped orient them 
to SIE’s vision and deepened their own understanding of the elements of effective science 
instruction. 

The revised Foundations course also emphasized the importance of science content in a number 
of ways. The Minds of Our Own video clips illustrated that simply doing hands-on activities is 
not sufficient to develop students’ understanding of important science concepts. This idea came 
up again later in the session, during the electromagnet investigation. As the group was 
discussing the discrepancies in the data they had collected, one teacher leader asked, “Does it 
matter who’s right as long as [students] support their claims with evidence and are being 
problem solvers?” The facilitators took advantage of this opportunity to point out that one 



Banilower, E., Cohen, K., Pasley, J. & Weiss, I. (2008) Effective science instruction: What does research tell us? 
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reason for having students make their claims and evidence public is so discrepancies in data 
collection can be resolved. The facilitators noted that, “You can’t just let [the discrepancies] go. 
You’ve got to get the right answer,” explaining that the investigation lays the foundation for 
learning additional concepts in subsequent lessons. The message was clear: content matters. 

A logical next step for the SIE program is to make sure that all teacher leaders have the 
knowledge and tools needed to keep the focus of the sessions they lead on the science content. 
Encouraging the teacher leaders to attend the Module Enrichment Training was a smart move, as 
engaging them with the content storyline of their module would also help ensure a consistent 
focus on the science content in the Initial Module Trainings. Another resource SIE may want to 
make available to teacher leaders is a series of books published by NSTA entitled Stop Faking It! 
These books offer an accessible overview of the key concepts related to a variety of science 
topics. 

The AEG also provided teacher leaders with an opportunity to reflect on how the design of the 
Initial Module Training incorporates the elements of effective science instruction. The grade- 
level groups had rich discussions, with facilitators asking probing questions that encouraged 
teacher leaders to apply their new knowledge of the elements of effective science instruction to 
analyze the design of the Initial Module Training. Eor example, the importance of sense-making 
about both the content in the module and the content storyline of the module were discussed. 

This activity provided a good opportunity for teacher leaders to start thinking about the 
implications of what is known about effective science instruction for their work as professional 
development facilitators. However, the teacher leaders have varied backgrounds (in terms of 
science content knowledge, teaching experience, and extent to which they have progressed from 
mechanical to purposeful implementation of the modules in their own classrooms) and therefore 
will need different kinds and amounts of assistance. One way to help support the teacher leaders 
and increase the quality and consistency of the Initial Module Trainings would be to update the 
facilitation guides previously created to include detailed information about how and when to 
highlight the elements of effective instruction. These updates could include: 

• Providing a suggested agenda, including time estimates for each portion of the training; 

• Making it explicit when teachers should be experiencing the module training activities as 
learners much like their students will be doing (i.e., “learner hat”), and when teachers 
should be reflecting on the training activities as teachers, considering the implications for 
their implementation of the module (i.e., “teacher hat”); 

• Providing a rationale for design decisions (e.g.. Do Eesson 3 with teachers from start to 
finish because it is logistically challenging and the sense-making is essential to the 
success of the subsequent lessons.); 

• Highlighting the science content students are intended to learn as a result of each lesson; 
and 
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• Indicating which lessons would be good candidates for discussing the elements of 
effective instruction with participants, including a description of why each of those 
lessons would be a good choice. 

Another major component of the ALC was the introduction of the “conversation tool.” 
Recognizing the importance of ongoing leadership development and the need to support teacher 
leaders in their facilitation of Initial Module Trainings, SIE has developed a strategy for on-site 
support in which a facilitation coach will observe these trainings and provide feedback using this 
tool. The tool guides both the coach’s observation and the subsequent discussion, and focuses on 
four components of facilitation: ease in working with adult learners; ease in working as a team 
member/co-facilitator; competency with pedagogy and module delivery; and positive attitude 
about science. 

The teacher leaders were introduced to this tool through the use of a “fishbowl,” in which the 
ASSET staff who led the Eoundations course sat in the center of the room and were debriefed by 
another staff member, modeling for the teacher leaders how the tool would be used. During this 
session, the ASSET staff emphasized that the tool was to help them continuously improve their 
work and that it was non-evaluative. The way the tool was introduced helped put teacher leaders 
at ease and established a culture of being reflective practitioners. 

This strategy has strong potential for helping further develop the knowledge and skills of the 
teacher leaders. In addition, it provides a mechanism for the program to monitor the quality of 
Initial Module Trainings. If the program decides to update its facilitator guides for the Initial 
Module Trainings, the conversation tool could also be used to further the teacher leaders’ 
understanding of the design rationale for these sessions. 

Overall, the AEG appears to have been successful at pushing the teacher leaders to take on the 
role of facilitator and increasing their comfort with switching from “teacher hat” to “facilitator 
hat.” The conversation tool activity, in particular, highlighted the importance of thinking about 
design decisions, general facilitation moves, and working with a co-facilitator. Eor example, the 
debriefing pointed to ways to curb sidebar conversations during a training while still honoring 
participants’ contributions to discussions. It also provided guidance on what to discuss with a 
co-facilitator prior to implementing a training together (e.g., discussing whether you are 
comfortable with the co-facilitator jumping in during your facilitation). 

Teacher leaders were also encouraged to think as facilitators when they were asked to identify 
the ways in which the elements of effective instruction could be highlighted for teachers in the 
Initial Module Training. The facilitator of one small group noted, “We tend to go right back to 
the classroom because it’s about the students, but we’re going to try to stay in the facilitator role 
[during this discussion].” This task forced participants to set aside their “teacher hat” and take 
on the perspective of a facilitator, thinking about how to engage teachers with important ideas 
about teaching and learning. 
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Development of Teacher Leaders: Leaders of Science Education Reform in 
Their Districts 

In addition to preparing teacher leaders for their role as facilitators of Initial Module Trainings, 
the ALC focused on strengthening the participants’ leadership skills so they could act as 
advocates for science education improvement in their schools and districts. For this goal, the 
ALC had two main objectives: 

1 . Helping the teacher leaders understand the different ways leadership can be exhibited; 
and 

2. Deepening teacher leaders’ understanding of factors that affect system-wide science 
education improvement. 

Helping the teacher leaders understand the different ways leadership can be exhibited was 
addressed in a number of ways throughout the conference: the presentation and discussion of 
attributes of famous leaders; the administration and discussion of results of a survey designed to 
assess the teachers’ natural leadership qualities; and the sharing of leadership successes teacher 
leaders have had at their schools. 

These activities appeared to be successful at helping the teacher leaders understand that there are 
different styles of leadership, and that leadership can occur on different scales (e.g., leadership 
can be exhibited in a school, district, state, or nation). The ALC also appeared successful at 
alleviating the teacher leaders’ concerns about their own leadership roles, helping them 
understand that they could contribute leadership in a number of different ways. In particular, the 
presentations by some of the more experienced teacher leaders as to how they have worked to 
improve science education beyond their own classrooms gave their colleagues an opportunity to 
see how small steps can affect change. 

The SIE program may want to consider ways to foster ongoing dialogue among the teacher 
leaders, both for the sharing of ideas for improving science education and to provide a support 
network among the teacher leaders. By sharing concerns and successes, the teacher leaders will 
continue to grow in their ability to act as science advocates in their schools and districts. One 
possibility is to hold a series of face-to-face meetings to allow for this cross-pollination of ideas. 
Another possibility is to utilize a web-based tool (e.g., listserv, discussion board). There are pros 
and cons to both approaches; for example the face-to-face meetings are more costly, while it can 
be difficult to sustain the conversation with a web-based approach given how busy teachers 
leaders are. Regardless of the mode, on-going discussion and support would likely be useful to 
the teacher leaders. 

The main activity for deepening teacher leaders’ understanding of factors that affect system- wide 
science education improvement was the Building Systems for Quality Teaching and Learning in 
Science game. Nearly the entire second day of the ALC was devoted to this game, with about 
equal amounts of time spent playing and discussing the game. During the game, groups of 
teacher leaders attempted to guide a fictional school, and its faculty, through the process of 
improving its science program. The teacher leaders had to consider issues such as the alignment 
of standards, curriculum, assessments, and instructional materials; building a common vision for 
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the school’s science program; the professional development needs of teachers; garnering support 
of stakeholders; and dealing with people resistant to change. The groups could spend resources 
to engage the teachers and administrators at the school in different types of activities (e.g., 
workshops on science content, observing each others’ classes, examining results from science 
assessments). The game was “won” when all of the teachers in the fictional school were brought 
on board and had reached a sufficient level of expertise. 

Overall, this game was very engaging to the teacher leaders. It raised a number of critical issues 
regarding the process of school-wide change, and provided a simulation environment in which 
the teacher leaders could try different strategies and observe the results. However, a number of 
the teacher leaders pointed out that they are not in a position in their schools to make the types of 
decisions involved in the game. Consequently, these teacher leaders struggled to see the 
relevance of the game. The game may have been more effective for these teacher leaders if their 
school principals had been involved. 

Another issue that arose was the competitiveness of the game. On the one hand, a healthy rivalry 
among the groups can help keep them focused and on-task. On the other hand, an overcharged 
race to the end can result in the messages being lost. Because the game allowed each team 
plenty of “resources” for their school, teams were able to simply try different combinations of 
activities quickly until they hit upon one that worked without being as thoughtful and deliberate 
as one needs to be when attempting to enact change in a real school. Finding the right balance is 
tricky, and may be different for different groups of participants. The program’s decision to stop 
the game process at different points and reflect on the issues was a good one, as it allowed the 
facilitators to re-direct participants as necessary. Doing so also allowed the facilitators to 
highlight and reinforce the key messages of the game. 

A next step the program may want to consider is providing the teacher leaders with tools that 
they can use to begin implementing what they learned from the game at their schools, much like 
the way the program supports schools in implementing the Showcase of Student Learning. 
Providing such tools would, if the tools were well designed, increase the likelihood that the 
teacher leaders will be successful advocates for science education improvement in their schools 
and districts. 
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Impact of SIE on Teachers and Teaching 



This section of the report focuses on how the SIE program has impacted teachers, including their 
preparedness to use their SIE-provided module and their classroom teaching. Data come from 
teacher and principal questionnaires and interviews, as well as classroom observations of module 
implementation. 



Teacher Preparedness 

One of the primary goals of the SIE program is to prepare teachers to use the SIE-provided 
modules. All interviewed teachers, across all cohorts, shared that they felt well prepared to teach 
the modules in their classrooms as a result of participating in the SIE professional development. 
As two teachers said: 

I felt ready to go. [The SIE professional development] made me feel comfortable and at 
ease with what to teach. 

[The professional development] lets you know just what materials you are going to need 
so you have a chance to get everything ready, and in the training they let you know, 

“This is what you need, ” and “This is what is in the kit. ” 

In addition to feeling comfortable with the materials and activities in the modules, teachers need 
to know how to enact the pedagogical approaches promoted by the program. The post- 
professional development questionnaire included a series of items aimed at assessing teachers’ 
perceptions of their preparedness to use the teaching strategies promoted by SIE (e.g., the 
inquiry-based teaching strategies embedded in the SIE module, science notebooks, a learning 
cycle). To assess impacts, teachers were asked about both their current preparedness and their 
preparedness prior to the professional development session. This “retrospective pre” approach is 
useful when respondents are likely to change their perceptions of initial knowledge/preparedness 
as they learn more about the topic (i.e., they didn’t realize how much/little they knew about a 
topic until after their participation in the program). 

Responses to these items were combined into a composite variable (called “perceptions of 

1 ^ 

pedagogical preparedness”) to reduce the unreliability associated with individual survey items. 
Each composite has a minimum possible score of 0 and a maximum possible score of 100. A 
score of 0 would indicate that a teacher selected the lowest response option for each item in the 
composite, whereas a score of 100 would indicate that a teacher selected the highest response 
option for each item. 

These longitudinal data have a nested structure, with time points nested within individual 
teachers. Statistical techniques that do not account for such nested data structures can lead to 



Definitions of this and other composites described in this report, a description of how the composites were 
created, and reliability information are included in Appendix B. 
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incorrect estimates of the relationship between independent factors and the outcome. 

Hierarchical regression modeling*^ is an appropriate technique for analyzing nested data and was 
used to examine trends in teachers’ composite scores. 

Across all workshop types, there were significant increases in teachers’ perceptions of their 
pedagogical preparedness. (See Figure 6.) The mean composite scores for Cohort 1 teachers 
attending a Module Enrichment Training for their first module increased from 53 to 77, a large 
effect of 1.21 standard deviations. Similarly, scores of Cohort 2 teachers who attended an Initial 
Module Training for their second module increased from a mean of 41 to a mean of 77, a very 
large effect of 1.75 standard deviations. Cohort 3 teachers’ scores increased from 34 points to 76 
points as a result of the Initial Module Training; this 42-point increase corresponds to a very 
large effect of 2.09 standard deviations. 

The difference in retrospective pre scores among the three cohorts is likely due to the fact that 
Cohort 1 and 2 teachers have been in the program longer and have attended trainings in previous 
years. Despite the differences in pre-workshop scores, teachers’ perceptions of pedagogical 
preparedness after attending their respective workshops, across all cohorts, were very similar. 



Teacher Perceptions of Their Pedagogicai Preparedness* 




Cohort 1 Module Cohort 2 Initial PD, 2nd Cohort 3 Initial PD, 1st 
Enrichment, 1st Module Module Module 

□ Retro Pre-PD n Post-PD 

* Teacher scores changed significantly over time, p < 0.05 (HLM) 

Figure 6 



The post-professional development questionnaire also asked teachers about their understanding 
of the student learning goals in the SIE module, the science content in the SIE module at a 
deeper level than what students are expected to learn, and ideas (either correct or incorrect) that 



Bryk, A.S. & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. 
Newbury Park, CA: Sage Publications. 
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students are likely to have about the eontent prior to instruetion. These items were eombined 
into a composite titled “perceptions of pedagogical content knowledge.” 

The analysis indicates that teachers’ perceptions of their pedagogical content knowledge were 
significantly greater after the SIE professional development. (See Figure 7.) Scores for Cohort 
1 teachers attending Module Enrichment Training increased over 20 points, a large effect of 1.04 
standard deviations. The 27-point increase for Cohort 2 teachers attending their Initial Module 
Training for their second module corresponds to a large effect of 1.34 standard deviations; the 
43-point increase in scores for Cohort 3 teachers corresponds to a very large effect of 2.09 
standard deviations. 



Teacher Perceptions of Their Pedagogicai Content 
Knowiedge* 



(fl 




Q. 

Cohort 1 Module Cohort 2 Initial PD, 2nd Cohort 3 Initial PD, 1st 
Enrichment, 1st Module Module Module 



□ Retro Pre-PD D Post-PD 

* Teacher scores changed significantly over time, p < 0.05 (HLM) 

Figure 7 



In interviews, teachers indicated that the Initial Module Training helped them understand the 
science content in their module. In the words of two teachers: 

It was nice that they went through and could explain the why of some of the experiments, 
which the teacher’s book doesn ’t really explain. The book is very, very limited at explaining. 
It was good because the presenters gave us notes to make sure the kids get the big idea of the 
lesson to get the concept that goes with the activity. It’s good to understand the concept 
they ’re supposed to be learning. 

I definitely came away understanding the concepts quite well and feeling that I was prepared 
to explain the concepts to the kids. 
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Module Implementation 

In Year One, SIE offered support for 1 1 different modules, and in Year Two SIE introduced a 
second set of 1 1 modules. During their first year in the program, each cohort selects one module 
to implement at each grade level. In their second year, each cohort adds a second module. No 
new modules were added in Year Three. 

As part of the student achievement study (described more fully later in this report), teachers 
complete a “module use” survey that asked them to indicate how much instructional time was 
devoted to each module they taught, and the extent to which they supplemented the module with 
other instructional materials. As can be seen in Table 11, teachers are basing substantial portions 
of their science instruction on the SIE-provided modules, with the average ranging from 83 to 95 
percent of their instructional time depending on the grade level and module. 



Table 11 



Percent of Instructional 



Time based on Module 





Cohort 1 


Cohort 2 


Cohort 3^ 


N 


Mean 


Standard 

Deviation 


N 


Mean 


Standard 

Deviation 


N 


Mean 


Standard 

Deviation 


Grade 3 




















Chemical Tests 


27 


95.33 


10.36 


23 


94.13 


8.21 


— 


— 


■ 


Plant Growth and 




















Development 


75 


85.57 


14.77 


85 


85.32 


14.39 


— 


— 


— 


Rocks and Minerals 


66 


91.17 


9.31 


87 


92.41 


10.13 


17 


92.12 


12.50 


Grade 4 




















Electric Circuits 


80 


92.83 


9.36 


100 


91.77 


12.70 


16 


95.31 


6.45 


Human Body 


34 


93.59 


9.80 


54 


83.26 


21.75 


— 


— 


— 


Grade 5 




















Ecosystems 


29 


88.97 


12.40 


46 


86.59 


16.16 


— 


— 


— 


Environments 


35 


88.20 


19.99 


24 


88.96 


19.39 


— 


— 


— 


Levers and Pulleys 


33 


89.52 


13.90 


35 


94.51 


7.48 


— 


— 


— 


Motion and Design 


28 


90.93 


11.55 


43 


85.35 


19.38 


20 


92.75 


12.62 


Grade 6 




















Experiments with Plants 


17 


75.24 


17.54 


13 


86.85 


16.46 


— 


— 


— 


Magnets and Motors 


11 


92.18 


8.68 


11 


89.09 


11.79 


— 


— 


— 


Mixtures and Solutions 


32 


91.47 


11.03 


18 


92.50 


12.75 


— 


— 


— 


Variables 


20 


89.95 


11.99 


24 


89.17 


18.55 


— 


— 


— 



Cohort 3 teachers had only a subset of modules available to them in their first year in the program. In addition, in order 
to maintain teacher anonymity, data are not shown for modules that only a small number of teachers implemented. 



In interviews. Cohort 1 and 2 teachers who had taught a module at least twice were asked to 
reflect on how their teaching had changed since the first time they taught a module. All of these 
interviewees indicated that their use of these modules had changed for the better as preparation 
for the lessons had become easier. These teachers noted that they were able to set up the 



Data were collected about the 13 modules that were a focus of the student achievement study. Of the 22 modules 
supported by SIE, 6 are for grades K-2 and 3 for grades 3-6 focus mainly on science process skills; these modules 
were not included in the student achievement study. 
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activities more quickly and to anticipate issues that might arise during instruction. As three 
teachers said: 

It’s quicker and easier set up. I know what I need to do for this lesson. I know what 
works best. Just getting all the supplies ready before we start. It gives me a better time 
frame and I know how to plan. 

I know what turned out [well] and what failed and I try to adapt to that. I feel that since 
I’ve taught it already, by the third time, I’m not as dependent on my notes. I’m more 
familiar with it. It becomes easier each time. 

You are more familiar with it and more on top of it. You understand where it’s going and 
the important aspects. When doing it cold, it’s not having the whole view of the total 
organization of the unit. The second time was much smoother. 

Cohort 1 and 2 teachers also indicated that they had less instructional time to implement the 
modules this year either because they had to return the module earlier than the previous year 
and/or because they were carving out instructional time to prepare for the PSSAs. Roughly half 
of the interviewed teachers said they were still able to cover all the module lessons, but that they 
spent less time on each lesson. As one teacher stated: 

We pushed [the module] along quicker this year, because six weeks is a hard time to give 
up when we have the PSSAs. ..I really followed the outline they gave me. . . We covered it 
all, but did it a little faster. 

The remaining half of the interviewed teachers reported that they ended up covering less of the 
module because of the time crunch. In the words of two teachers: 

Well sometimes it’s hard to get to all the lessons. They now make us do it in a shorter 
amount of time. If the kit has to be back in a week, I would know which ones to skip. 

This year we didn’t get to complete [the module] because we ran out of time and had to 
send it back. If we could have kept the kit longer, I would have finished. It would be nice 
to have more time with the kits, to not feel rushed. 

The program design calls for schools to incorporate a third science module in their fourth year in 
the program. In interviews, Cohort I teachers were asked about the feasibility of implementing a 
third module. Teachers’ opinions were mixed, with about half saying they would not have 
enough instructional time to fit another module in and the others saying they would make it 
work. SIE should consider how it might help teachers manage the implementation of three 
modules, perhaps by re-examining the module delivery and pick-up schedule, and perhaps by 
providing suggestions to teachers on which lessons in a module are critical and which can be 
omitted if they are pressed for time. 

One factor that may influence teachers’ implementation of the modules is their perceptions of 
principal support. SIE teachers were asked a series of items on the end-of-year questionnaire 
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about the extent to which their principal supported their science teaching and the SIE program. 
These items were combined into a composite titled, “perceptions of principal support.” As can 
be seen in Figure 8, scores on this composite were fairly high across the three cohorts, indicating 
that teachers felt that their principals support their implementation of the SIE program. Cohort 
2 teachers’ perceptions changed significantly from their first to second year in the program, 
though the decrease is relatively small (an effect size of 0.24 standard deviations). 



Teacher Perceptions of Principal Support 




□ 2007-08 □ 2008-09 

*Cohort 2 teachers had significantly different perceptions of principal support in Year Two than in Yeai* 
One, p < 0.05 (Independent samples t-test) 

Figure 8 



Most interviewed teachers also reported that they felt supported by their administration in 
implementing the modules. In some cases, this support was in the form of dedicated blocks of 
time for science instruction being set aside and protected from competing priorities. In other 
cases, principals provided funds for teachers to purchase materials needed to implement 
extensions to the module lessons. 

Teachers and principals were asked on questionnaires about other factors that may have affected, 
positively or negatively, teachers’ use of the modules. Table 12 and Table 13 show teacher and 
principal responses to these questions from this and last year. 

Nearly all responding teachers and principals in both years thought that the training provided by 
SIE facilitated the use of the modules. In addition, most respondents indicated that the support 
of fellow teachers and the SOS facilitated use of the modules. Interestingly, although the SIE 
program does not have any formal mechanisms for assisting teachers outside of the workshops. 



This set of items was added to the end-of-year questionnaire in 2007-08; thus there is no baseline data for Cohort 
1 on this factor. 
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approximately one-quarter of the teachers and more than half of principals indicated that teachers 
received extra assistance that helped them implement their module. (Teachers’ and principals’ 
responses to these questionnaire items disaggregated by cohort can be found in Appendix A.) In 
addition, the percentage of teachers indicating that their own science background and their SOS 
facilitated their use of the modules increased significantly from last year to this year, though the 
changes were fairly small (an effect size of 0. 12 for each change). 



Table 12 



Teacher Opinions of Factors Affecting 
Their Use of the SIE-Provided Module, by Year 





Percent of Responding 
Teachers Agreeing' 






2007-08 
(N = 1,748) 


2008-09 
(N = 1,561) 


Effect 

Size 


Factors Facilitating Use 

The training I received from SIE made it easier for me to use the 
modules. 


95 


94 




Other teachers in my school provided a support system for use of the 
modules. 


86 


88 




My own science background was helpful when I was teaching from 
the module.* 


77 


82 


0.12 


My SOS facilitated my use of the modules.* 


73 


78 


0.12 


I received assistance from SIE outside of the workshop that helped 
me to use the modules. 


26 


25 




Factors Inhibiting Use 

The pressures to teach mathematics and/or reading inhibited my use 
of the modules. 


51 


51 




The amount of time required to prepare for instruction with the SIE 
modules was problematic.* 


55 


46 


-0.18 


There was not enough instructional time for science to effectively use 
the SIE modules.* 


52 


45 


-0.14 


I am not able to cover all of the science topics I am supposed to 
because of the time it took to implement an SIE module. 


43 


42 


— 


I did not have the modules for long enough to use as much of them as 
I wanted to.* 


29 


41 


0.25 


The amount of time I was required to be out of the classroom was 
problematic.* 


26 


14 


-0.30 


My lack of experience in science made it more difficult for me to 
teach from the module.* 


15 


10 


-0.15 



Includes those indicating “Agree” and “Strongly Agree” on a four-point scale of 1 “Strongly Disagree” to 4 “Strongly 



Agree.” 

* The percentage of teachers agreeing with statements at the end of the 2008-09 school year was significantly different 
than at the end of the 2007-08 school year (two-tailed z-test, p < 0.05). 



When comparing percents, the effect size is calculated using the difference between the arcsine transformation of 
the percents. 
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Table 13 



Principal Opinions of Factors Affecting 
Teacher Implementation of the SIE-Provided Modules, by Year 





Percent of Responding 
Principals Agreeing^’* 




2007-08 
(N = 93) 


2008-09 
(N = 100) 


Factors Facilitating Use 

The training teachers received from SIE made it easier for them to use the modules 


99 


99 


Teachers provided each other with a support system for using the modules 


98 


96 


Our Support on Site (SOS) facilitated teachers’ use of the modules 


85 


93 


Teachers received assistance from SIE outside of the workshop that helped them to use the 
modules 


61 


58 


Factors Inhibiting Use 

The amount of time teachers were required to be out of the classroom was problematic 


57 


40 


The time of year that teachers received their modules was problematic 


— 


38 


There was not enough science instructional time for teachers to complete the SIE modules 


26 


38 


Teachers did not have the modules for enough time to complete all of the lessons 


19 


37 


The pressures to teach mathematics and/or reading inhibited teachers’ use of the modules 


38 


33 


Teachers were not able to cover all of the science topics they were supposed to because of 
the time it took to implement an SIE module 


30 


25 


The amount of time required to prepare for instruction with the SIE modules was 
problematic for teachers 


23 


22 


Too much time elapsed between the module training and module delivery 


— 


18 


Teachers’ lack of science content knowledge made it more difficult for them to teach from 
the modules 


12 


16 


Teachers’ lack of experience in science made it more difficult for them to teach from the 
modules 


12 


13 



Includes those indicating “Agree” and “Strongly Agree” on a four-point scale of 1 “Strongly Disagree” to 4 “Strongly 
Agree.” 



* The percentage of principals agreeing to each statement in 2008-09 is not significantly different than in 2007-08, p > 
0.05 (two-tailed z-test). 



In terms of inhibitors, time remained the major concern of both teachers and principals. Similar 
to last year, about half of the responding teachers indicated that pressures to teach mathematics 
and/or reading, the amount of time required to prepare for instruction with the SIE module, and 
the lack of instructional time for science inhibited their use of the module. There was, however, 
a small decrease in the percentage of teachers indicating these latter two issues were problematic 
this year. The amount of time teachers are required to be out of the classroom was also a 
concern of 4 out of 10 principals; only 14 percent of teachers indicated that this issue was a 
concern, a significant decrease from the 26 percent who identified it as a concern last year (a 
small effect of -0.30). 

One concern that has grown in magnitude among teachers this year was the amount of time they 
have with the module. In 2008-09, 41 percent of teachers indicated that this factor inhibited 
their use of the SIE modules, up from 29 percent in 2007-08 (a small effect of 0.25). This 
finding is likely due to the module delivery and pick-up schedule this year, which provided 
teachers less time to implement the module than last year. 
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In interviews, teachers and principals elaborated on factors that inhibited their use of the 
modules. As two teachers reported: 

Obviously, PSSA state testing. We have to spend more time in reading and math to get 
them ready. That’s the big one; that’s our big speed bump that we have to go over every 
year. 

We had a problem with the kits being taken from us too early. Some [of my colleagues] 
were not finished with the kit and it was due back before they were able to finish teaching 
it thoroughly. 

Finally, as described previously in this report, responses to this item highlight a tension between 
wanting instruction to go in depth on a smaller number of topics versus covering all of the 
content in the state science standards. Forty-three percent of responding teachers and 25 percent 
of responding principals expressed concern that teachers are not able to cover all of the science 
topics they are supposed to because of the instructional time it takes to implement an SIE 
module. In the words of two teachers: 

I do not think that students are getting the science content that they need to cover the PA 
state science standards. 

I find it difficult to explore the other areas of science for the state test because so much 
time is consumed with the SIE Program. I love the program, but feel that some of the 
repetitive material could be removed. 

As the program moves forward, it should consider ways of addressing this concern; e.g., 
identifying for teachers which module lessons they could skip or truncate when pressed for time. 
It would be imperative that trained program staff identify these lessons, not the teachers 
themselves, to ensure that the modifications to the modules to be purposeful and not exclude key 
concepts. 



Student Opportunity to Learn 

To gain insight into how the modules and the associated professional development were being 
translated into classrooms, members of the evaluation team observed a sample of 12 participants 
teaching a lesson from a module, 8 in Fall 2008 and 4 in Spring 2009. The Fall observations 
focused on Cohort 1 teachers, each of whom had participated in the Initial Module Training and 
were implementing the module for the third time. The four modules from which lessons were 
observed were: Electric Circuits', Levers and Pulleys', Mixtures and Solutions', and Motion and 
Design. In the Spring of 2009, four Cohort 3 teachers were observed teaching their first SIE 
module for the first time. The four modules from which lessons were observed in the Spring 
were: Changes', Levers and Pulleys', Mixtures and Solutions', and Motion and Design. 

The classroom observations focused on the quality of the instruction in terms of opportunity for 
students to deepen their understanding of the intended content. Interviews after the observations 
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were used to gather information on the teachers’ experiences with the module and with SIE 
professional development. As was the case in observations of the SIE professional development, 
the framework HRI used for assessing student opportunity to learn is based on research on how 
people learn. Eollowing is an overview of the analysis of the observed lessons in relation to this 
framework, as well as an analysis of factors that may have affected student opportunity to 
learn. 

Extent to Which the Classroom Culture/Learning Environment Facilitated Student 
Opportunity to Learn 

Eeaming is facilitated by a classroom culture that encourages student participation and exhibits 
high expectations for all students. The classroom culture was a strength of most the observed 
lessons; a majority of the lessons had supportive learning environments characterized by a 
collegial atmosphere that encouraged student participation. Eor example, in a 6* grade Mixtures 
and Solutions lesson, the teacher used multiple strategies to involve all students: 

At the start of the lesson, the teacher set the tone that all students were expected to participate in the lesson 
by saying to the class, “We are going to do an experiment today, but first I want to address something. 
Yesterday, when I was asking questions and nobody was answering, that bothered me. Let’s make sure we 
talk and participate.” During the activity, the teacher circulated around the room, making sure all students 
worked with the materials. The teacher also made sure to call on different students during the class 
discussion, which got many students involved in the conversation. The teacher’s efforts seemed to pay off, 
as students appeared comfortable providing responses to the teacher’s prompts. For example, when the 
teacher was wrapping up the lesson, many students shared their ideas without prompting by the teacher. 

The culture of one lesson interfered with students’ opportunity to learn; this lesson appeared to 
be the exception and not the rule. In this classroom, the teacher struggled with classroom 
management and the students were subsequently off task, which negatively impacted the 
opportunity to learn. 

Extent to Which All Students Were Intellectually Engaged with the Targeted Ideas in the 
Lesson 

In order for students to learn new ideas, they must be intellectually engaged with those ideas, 
thinking and talking about the science content, and responsible for doing the intellectual work. 

In other words, instruction must be “minds-on,” not just “hands-on.” In addition, the science 
content with which they are engaged must be accurate. Of the 12 lessons observed, in only half 
were students engaged with mostly accurate content; the other half had major inaccuracies that 
likely prevented students from learning the targeted science ideas. For example, in one lesson 
students were allowed to use “water” and “liquid” interchangeably, which might lead the 
students to believe that water is a different substance than ice or water vapor. In another lesson, 
the teacher interchanged the words “force” and “energy” a number of times, which likely caused 
the students to think the two are synonymous. (Research on student ideas about science has 
found that this incorrect idea is resistant to change.) Interestingly, all of these teachers indicated 
that they had a fairly good understanding of the content in the module they were teaching. Even 
those lessons taught by teachers who had attended Module Enrichment contained substantial 



More information on the Fall 2008 observations can be found in; Fulp, S.L. & Banilower, E.R. (2008). Fall 2008 
Classroom Observation Study. Chapel Hill, NC: Horizon Research, Inc. 
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content errors, indicating that teachers need additional content support, even though they may 
not recognize that need themselves. 

In addition to looking at the accuracy of content, observers looked at other aspects of intellectual 
engagement with the targeted content. For example, observers noted whether students seemed to 
understand what they were doing and why, were talking about the targeted concepts, and were 
responsible for doing the intellectual work. In each of the observed lessons, students were 
actively participating in the activities and appeared to be enjoying themselves. However, only 2 
of the 12 observed classes provided a great deal of opportunity for students to intellectually 
engage with the learning goals; in these lessons, students spent time thinking, talking, and 
writing about the targeted concepts. In one of these lessons, the teacher used a whole-class 
discussion to focus students on the important science ideas: 

The purpose of a fifth grade Motion and Design lesson was for students to understand that engineers 
develop, modify, and improve designs to meet specific requirements, and that design requirements specify 
how a product must perform. Students were able to share their thoughts on force and motion and the 
design of their vehicles with the whole class during the class discussions. For example, one group shared 
that their vehicle was originally too top-heavy and would fall over when they pushed it, so they made 
modifications to decrease the weight. Another group discussed their observation that the location on the 
vehicle where the force was applied had more of an impact on the distance it traveled than the actual design 
of the vehicle. 

In the other lesson, the teacher used note-booking strategies to intellectually engage the students 
with the targeted science content: 

In a sixth grade Mixtures and Solutions lesson, students were to begin to develop the idea that after the 
evaporation of a liquid, some solids form crystals which have distinctive patterns. Students made 
observations about salt crystals evaporated from a salt-water solution during the lesson. Students wrote 
about the properties of crystals in their science journals. They each described what they observed in words, 
and then made a scientific drawing on which they labeled the observed properties of the salt crystals. 

However, in the other observed lessons intellectual engagement with the targeted ideas was low. 
In several lessons, the students were not attempting to answer an important science question and 
were simply going through the motions of the activity rather than conducting a purposeful 
inquiry. For example, the goal of an observed Levers and Pulleys lesson was for students to 
learn about the three classes of levers. The students in this lesson spent a disproportionate 
amount of time manipulating the materials instead of thinking and talking about the targeted 
concepts: 



These fifth grade students had learned about a Class 1 lever in a previous lesson. In this lesson they 
explored different classes of levers by moving the fulcrum, effort, and load. The majority of class time was 
spent allowing students to build the different classes of levers. Students were not required to take 
measurements, make observations, or record the different configurations in their notebooks. They simply 
were required to place the load, effort, and fulcrum in each of the different positions without any 
mechanism to focus students on how and for what reasons one would create and use the different classes of 
levers. 

A second way intellectual engagement was limited was by students collecting inaccurate data, 
hindering their ability to draw scientifically appropriate conclusions from the data. For example: 
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In a different lesson from the Levers and Pulleys module, students were supposed to learn that the greater 
number of ropes supporting a load, the smaller the effort that is needed to lift it. Students worked in groups 
for half an hour to create four pulley systems, gathering data on each. Data collection was not done 
carefully. Students were observed using the spring scales incorrectly, resulting in inaccurate data. In 
several groups, the rope came off of the pulley track, catching between the pulley and the pulley holder, 
which also impacted measurements. The teacher moved from group to group, helping students correct data 
collection mistakes, but she did not catch many of them. As a result, the data collected did not support the 
accepted scientific idea that the effort required decreases as the number of supporting ropes increases. 

Third, in several cases students were observed engaging with just a subset of the targeted 
concepts in the lessons. For example, in a Motion and Design lesson the teacher successfully 
engaged students with the idea that a force is a push or pull, but not that a force can change the 
speed of an object. 

These data indicate a need to help teachers understand all of the targeted ideas for each lesson in 
a module, both those that are explicit in the module and those that are implicit (i.e., concepts that 
are embedded in a lesson, but which the teachers’ guide does not explicitly tell teachers to 
emphasize). Teachers would also benefit from a deeper understanding of how each activity 
contributes to the development of the science ideas. In addition, teachers would likely benefit 
from suggestions on how to help students attend to the targeted ideas during instruction. The 
focus questions provided to teachers during the Initial Module Trainings are a good start, but 
given the difficulties observed in focusing students on the concepts behind the activities, 
including additional tools, such as specific questions teachers might ask to gauge student 
thinking during the activities, should also prove beneficial. 

Extent to Which Students Used Evidence to Support and Critique Conclusions 

Understanding how scientific knowledge is generated is an important aspect of learning science. 
Students need to know that science involves the drawing of conclusions based on evidence, not 
the memorization of a series of facts. The SIE-supported modules are designed with the 
principle that students will construct scientific ideas based on evidence gathered from first-hand 
experiences. Accordingly, the extent to which students used evidence to support and critique 
conclusions was examined in the observed lessons. 

In only 1 of the 12 observed lessons did the teacher emphasize drawing and supporting 
conclusions with evidence. In this lesson from the Motion and Design module, students were 
learning that the greater the force pulling on a vehicle, the greater the change in speed of the 
vehicle. 

Students connected a string to cars, hung the string off the table with washers on it. Students collected data 
on the speed of the car with the addition of washers. After the students had completed the activity in the 
lesson, the teacher led a class discussion about their results. During the discussion, the teacher asked 
questions such as, “What happened to the speed from two washers to four washers?” The teacher then used 
the student response that “the speed increased” as a basis for asking them what they noticed about what was 
happening as the number of washers increased, and connected the increasing speed to an increase in the 
pull on the vehicle. In this way, the conclusions from the activity were generated from the data the students 
collected about the speed of their vehicles. 

It is important to note that although the teacher accurately modeled the building of scientific 
knowledge from evidence, the conclusion students were drawing about the relationship between 
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mass and speed was incorrect. Although the activity focuses students on the speed of the car by 
having them rank the speed in each trial, force is related to the rate of change in speed, not speed 
itself. 

In the majority of observed lessons, the development of scientific knowledge was not portrayed 
as a process based on drawing and critiquing conclusions using evidence. In several observed 
lessons, students collected data, but did not use those data to draw conclusions. For example, in 
a Mixtures and Solutions lesson, students made observations about what they saw when they 
mixed calcium chloride, baking soda, and water in a zip bag. Students were asked to report what 
they had observed but were never asked to come to a conclusion about whether a chemical 
reaction had occurred (the targeted concept for the lesson). Similarly, in a lesson from the 
Electric Circuits module, students constructed circuits with a diode. Students drew diagrams of 
the circuits that worked in their science notebooks, but these diagrams were not used as evidence 
to support any conclusions about the role of a diode in a circuit. 

To help teachers develop a better understanding of the process by which scientific knowledge is 
generated, it will be important that the professional development they receive from the SIE 
program: (1) consistently and accurately model this process; and (2) explicitly reflect on the 
process with teachers after they work through the activities as learners. 

Extent to Which Students Were Given the Opportunity to “Make- Sense” of the Targeted 
Ideas 

It is clearly important for students to be engaged with activities that are designed to develop their 
conceptual understanding and to use evidence in constructing those ideas. It is equally important 
that students be given opportunities to make sense of the targeted science concepts. Without this 
sense-making, students may simply do the activities without ever understanding the science ideas 
and how they connect to previously learned ideas. 

The observed lessons provided little opportunity for students to make sense of the targeted ideas. 
Many times this lack of sense-making was because the teacher modified the lesson. In a number 
of cases, it was due to the teacher modifying the wrap-up discussion by focusing the 
conversation simply on students’ observations rather than on the targeted science ideas and how 
the observations supported those ideas. 

In some cases, lessons engaged students with only a subset of the targeted concepts. As a result, 
sense-making occurred for some ideas but not for others. For example, in a lesson from the 
Mixtures and Solutions module, students mixed chemicals in order to learn that chemical 
reactions produce new substances that are different from the reactants, and that not all chemicals 
react when they are mixed. The lesson calls for students to test the precipitate from the final 
reaction with vinegar to determine if it is chalk. The teacher modified the lesson by having 
students skip this part of the lesson. Without the results of this test, students would not be able to 
determine if new substances were formed as a result of their experiment, one of the key learning 
goals of the lesson. 

There was some indication that the reason for these modifications was lack of teacher 
understanding of the purpose of the lesson. During the post-observation interviews, teachers 
were asked what they hoped students would learn from the lesson. Only 5 of the 12 teachers 
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stated goals that were aligned with the goals identified in the modules. Most of the teachers 
either mentioned only a subset of the targeted concepts and/or named goals that were not the 
same as those identified for the lesson. 



Finally, in some lessons sense-making could not occur because of content inaccuracies, which as 
described previously, were a major problem in several of the observed lessons. During the post- 
observation interviews, all of the teachers indicated that they believed their students “got” the 
targeted concepts of the lesson. These responses provide further evidence that the teachers did 
not have a solid understanding of what students are supposed to learn from the activities or how 
to judge whether students mastered the intended content. 

Overall Extent to Which the Lessons Provided Students with an Opportunity to Learn the 
Targeted Ideas 

All of the observed lessons included some elements of effective practice, though few “put it all 
together.” Strengths of the lessons included collegial interactions among students and respectful 
contact between the students and the teacher. Students engaged in many hands-on activities, but 
the amount of intellectual rigor required of the students in most lessons was generally low. In 
addition, many lessons lacked adequate sense-making of the targeted science ideas. These 
findings may be a result of the Initial Module Trainings focusing heavily on familiarizing 
teachers with the lesson activities, often with limited time for teacher reflection on the targeted 
science being taught through the module activities and how to help students understand that 
science. An increased focus on sense-making of the science content during the professional 
development would likely lead to an increase in teachers incorporating this element in their own 
instruction. 
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Impact of SIE on Students 



Although the main emphasis of the program’s activities is on teachers and their instruction, the 
ultimate goals of the program is to improve student attitudes toward and learning of science. The 
evaluation addressed this aspect more systematically through two activities, one aimed at 
assessing the impact of the program on student attitudes toward science, the second examining 
the program’s impact on student learning. 

To assess the impact of the program on student attitudes toward science, teachers were asked a 
series of items in the end-of-year questionnaire about their students’ attitudes toward science and 
the extent to which they attributed those student attitudes to the SIE program. As can be seen in 
Table 14, the vast majority of teachers indicated that students had positive attitudes for science, 
with over 9 in 10 agreeing that students were enthusiastic about science class, enjoyed learning 
about science, and wanted to learn more science. In addition, 70 percent of respondents 
indicated that their students liked science more than previous ones they had taught. In addition, 
teachers attributed these positive attitudes to the SIE program. (See Table 15.) 



Table 14 



Teacher Opinions about Students’ Attitudes toward Science 





Percent of Responding Teachers 
Agreeing^ 

(N = 1,520) 


Students... 




Were enthusiastic about science class 


97 


Enjoyed learning about science 


96 


Were interested in science 


95 


Wanted to learn more science 


93 


Were eager to explore science on their own, outside of the classroom 


85 


Liked science more than previous students I have taught 


70 



Includes those indicating “Agree” and “Strongly Agree” on a four-point scale of 1 “Strongly Disagree” to 4 “Strongly 



Agree.” 



Table 15 



Teacher Report of Impacts on Students 







Percent of “Agreeing” Teachers 
Attributing Impact to SIE 






Not at all 


To some 
extent 


To a great 
extent 


Students... 

Enjoyed learning about science 


1,440 


1 


34 


65 


Were interested in science 


1,416 


2 


39 


59 


Were enthusiastic about science class 


1,449 


1 


43 


56 


Liked science more than previous students I have taught 


1,032 


1 


52 


47 


Wanted to learn more science 


1,388 


1 


54 


45 


Were eager to explore science on their own, outside of the 
classroom 


1,270 


3 


68 


29 



The “N” represents the number of teachers out of the 1,520 included in Table 14 who agreed with each statement about 
their students. The percent of teachers attributing the impact to SIE is based on the number of teachers agreeing with the 
statement. 
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To examine the impact of the program on student learning, HRI, with the assistance of the 
Pennsylvania Department of Education (PDE), designed and implemented a student achievement 
study. Specifically, this study examined the relationship between teacher participation in SIE 
professional development and their implementation of the science modules and student 
achievement. Eor Year Three, the study was modified in several important ways from the 
student achievement studies implemented in the first two years of the program. Because of the 
increasing number of SIE-supported modules schools were implementing, it was no longer 
feasible to administer an assessment before and after each module a teacher implemented; too 
much instructional time would be lost due to testing. Consequently, a grade- specific assessment 
was developed for each grade 3-6 that covers all of the modules supported by SIE at that grade 
that focus on developing science conceptual understanding (modules that focus on developing 
science process skills were not included). 

This Year Three study design sought to answer the following questions: 

1 . Is there a relationship between the extent to which science instruction is based upon the 
SIE- supported modules and student achievement? 

2. Is there a relationship between teacher participation in SIE-provided professional 
development and student achievement? 

3. Are there gender or race/ethnicity differences in student achievement, and if so, does the 
extent to which instruction is based on the SIE-supported modules and/or teacher 
participation in SIE-provided professional development reduce those differences? 



Instrumentation 



HRI developed the four assessment scales that were used in this study, with each scale 
corresponding to the science modules at a grade level provided by the SIE program. The 
modules that were addressed by the assessment at each grade level were: 



Grade 3 



Grade 4 



Grade 5 



1. Chemical Tests (STC); 

2. Plant Growth and Development (STC); 

3. Rocks and Minerals (STC); 

4. Sound (STC); 

5. Structures of Eife (EOSS); 

6. Animal Studies (STC); 

7. Electric Circuits (STC); 

8. Human Body (EOSS); 

9. Eandforms (EOSS); 

10. Ecosystems (STC); 

11. Environments (EOSS); 
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12. Floating and Sinking (STC); 

13. Levers and Pulleys (FOSS); 

14. Motion and Design (STC); 

Grade 6 15. Experiments with Plants (STC); 

16. Food and Nutrition (FOSS); 

17. Magnets and Motors (STC); 

18. Mixtures and Solutions (FOSS); and 

19. Variables (FOSS). 

Because of the large number of students involved in the SIE program, and the timeframe in 
which results were needed, it was also decided to use only selected-response (i.e., multiple- 
choice) items rather than including open-ended or performance items. Eor each grade, HRI 
developed a pool of assessment items that covered the concepts included in both the modules and 
the Pennsylvania Science and Technology Standards. All items went through a stringent, 
internal review to ensure both alignment with the targeted content and language accessibility. 
When possible, cognitive interviews were conducted with elementary students to help ensure that 
students interpreted the items as intended, and that student responses were appropriate given 
their understanding of the content (i.e., those students who understood the content responded 
correctly, and those who did not understand the content selected an incorrect response). The 
items were then reviewed for content accuracy by Ph.D. scientists with expertise in the relevant 
topic area; any items with content issues were revised or removed from the item pool. HRI also 
sent each item pool to PDE for review and approval. These steps provide some assurance of 
content validity of the assessments. 

In addition, statistical analyses can be used to examine the validity and reliability of items. 

Eactor and dimensionality analyses help determine whether a set of items form a scale (i.e., a set 
of items that measure the same ability or trait, for example knowledge of levers and pulleys). 
These analyses resulted in the dropping of a small number of items. In addition. Differential 
Item Eunctioning (DIE) was used to examine whether items showed evidence of gender or 
race/ethnicity bias. No items had to be dropped due to bias concerns. Table 16 shows the 
number of items out of the original set for each scale that were retained for these analyses, as 
well as the reliability for each set of items. 



Table 16 

Assessment Scale Reliabilities 





Number of Items Retained 


Reliability 


3"^“ Grade 


29 of 30 


0.74 


4* Grade 


29 of 30 


0.77 


5* Grade 


37 of 40 


0.81 


6'" Grade 


34 of 40 


0.83 



Reliability ranges from 0 to 1 ; typically, a reliability > 0.60 is considered acceptable for assessments used to 
evaluate a program. 
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Race/ethnicity and gender data were also collected from students. Finally, teachers provided 
information about their classes, including the amount of instruction they provided on each of the 
topics on the assessment and the extent to which that instruction was based on an SIE-supported 
module. 



The Sample 

For the 2008-09 student achievement study, all schools participating in the SIE program that 
contain any of the grades 3-6 were asked to participate. Of these 137 schools, 99 (44 Cohort 1, 
42 Cohort 2, and 13 Cohort 3 schools) returned pre- and post-test data in time to be included in 
these analyses. The great majority of data returned were included in the analyses; however, data 
from a number of classes were excluded due to assessment administration errors (e.g., some 
teachers did not follow the instructions for distributing answer sheets, making it impossible to 
link student pre- and post- test data for those classes). Table 17 shows the number of classes that 
were expected to be part of the study for each assessment scale, as well as the number of classes 
included in the final analyses. 



Table 17 



Number of Classes in the Study, by Cohort 





Cohort 1 


Cohort 2 


Cohort 3 


Expected 


Included in 
Analyses 


Expected 


Included in 
Analyses 


Expected 


Included in 
Analyses 


Grade 


156 


126 


178 


149 


44 


41 


4* Grade 


167 


140 


174 


149 


48 


39 


5* Grade 


177 


169 


126 


110 


48 


45 


6* Grade 


140 


126 


98 


87 


20 


18 



Table 18 provides demographic information for the students included in these analyses. Overall, 
the classrooms contained the same proportions of females and males. Most students classified 
themselves as White; nearly all students indicated that English was their primary language. 
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Table 18 



Student Demographic Data 





Percent of Students 




3*^“ Grade 


4“* Grade 


5“* Grade 


6“* Grade 




(N = 5,086) 


(N = 5,631) 


(N = 5,686) 


(N = 3,934) 


Gender 










Female 


50 


50 


50 


49 


Male 


50 


50 


50 


51 


Race/Ethnicity^ 










American Indian/ Alaskan Native 


1 


0 


1 


1 


Asian 


1 


1 


1 


1 


Black/ African- American 


11 


10 


9 


12 


Hispanic/Latino 


2 


2 


3 


3 


Native Hawaiian/Other Pacific Islander 


0 


0 


0 


0 


White 


87 


87 


89 


86 


English Language Learner 


1 


1 


1 


2 



’ The total percent may add to more than 100 as students could select more than one category. 



Analysis and Results 

Item response theory (IRT) was used to compute difficulty and discrimination parameters for 
each item. These parameters were then used to calculate “true test scores” for each student, an 
IRT-based score that takes into account the relative difficulty of each item. Using the true test 
score removes the error associated with day-to-day fluctuations in student performance, and is a 
better estimate of student knowledge than a raw score such as number or percent correct. The 
true test scores were scaled to have a minimum of 0 and a maximum of 100. Table 19 shows the 
true test scores for each assessment scale. 



Table 19 
True Test Scores 





Minimum 


Maximum 


Mean 


Standard 

Deviation 


3'^“ Grade 










Pre-Test 


20.72 


79.05 


33.51 


8.62 


Post-Test 


20.72 


87.69 


43.11 


12.73 


4“' Grade 










Pre-Test 


23.34 


79.35 


40.57 


10.24 


Post-Test 


23.52 


89.05 


50.92 


12.71 


5"> Grade 










Pre-Test 


20.90 


86.22 


37.13 


10.49 


Post-Test 


20.90 


90.23 


46.07 


14.11 


6“' Grade 










Pre-Test 


23.58 


85.18 


49.26 


13.34 


Post-Test 


23.58 


89.91 


55.50 


15.00 



The student assessment data have a nested structure, with test administration (pre-test and post- 
test) nested within students nested within classes. Statistical techniques that do not account for 
potential grouping effects (e.g., all students in a class having common instructional experiences) 
in nested data structures can lead to incorrect estimates of the relationship between independent 
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factors and the outcome. Hierarchical (multilevel) regression modeling was used to examine 
student assessment scores. Results for each model follow, organized by research question. 
Regression coefficients and standard errors are presented in Appendix C. 

> Is there a relationship between the extent to which science instruction is based upon the 
SIE- supported modules and student achievement? 

To answer this question, HRI used data from the Module Use Surveys administered to teachers at 
the middle and end of the school year. This survey asked teachers which SIE-supported modules 
they implemented, how many science lessons they taught on the assessed topics, and what 
proportion of instructional time was based on the modules. These data were used to create three 
variables for the analyses that summarize their instruction on the topics covered by the 
assessment: the number of assessed topics for which the teacher provided instruction; the amount 
of instructional time using SIE-supported modules; and the amount of instructional time using 
non-SIE materials. As can be seen in Table 20, the amount of science instruction on the tested 
topics was relatively low, ranging from an average of about 18 hours in 3'^'^ grade to roughly 36 
hours in 6* grade. There was a fair amount of variation in instructional time among classes 
though, as evidenced by the relatively large standard deviations. In addition, the high maximum 
values indicate that some teachers spent a great deal of time teaching the assessed topics. 



Table 20 

Science Instruction in SIE Classes 





Minimum 


Maximum 


Mean 


Standard 

Deviation 


3''“ Grade (N = 245) 










Number of Assessed Topics Taught 


0.00 


5.00 


2.28 


1.14 


Instructional Hours Using SIE-Supported Modules 


0.00 


76.50 


13.47 


10.70 


Instructional Hours Using Non-SIE Materials 


0.00 


90.30 


4.98 


9.62 


4“' Grade (N = 256) 










Number of Assessed Topics Taught 


0.00 


4.00 


2.49 


1.12 


Instructional Hours Using SIE-Supported Modules 


0.00 


75.83 


13.89 


12.23 


Instructional Hours Using Non-SIE Materials 


0.00 


75.00 


8.98 


12.51 


5“' Grade (N = 229) 










Number of Assessed Topics Taught 


0.00 


5.00 


2.89 


1.30 


Instructional Hours Using SIE-Supported Modules 


0.00 


78.25 


23.13 


15.29 


Instructional Hours Using Non-SIE Materials 


0.00 


65.25 


6.25 


10.88 


e*” Grade (N = 160) 










Number of Assessed Topics Taught 


0.00 


5.00 


2.36 


1.05 


Instructional Hours Using SIE-Supported Modules 


0.00 


85.00 


27.72 


18.37 


Instructional Hours Using Non-SIE Materials 


0.00 


88.00 


8.26 


15.54 



The regression models examined changes from pre- to post-test scores, and the extent to which 
the three variables characterizing teachers’ instruction related to those changes. The models also 
examined whether achievement gaps existed by gender and race/ethnicity, and if there was 
adequate class-level variation, the models investigated whether the instructional variables were 
related to changes in any existing gaps. Because the number of students classifying themselves 
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as any race/ethnic group other than White was small, student race/ethnicity data were collapsed 

90 

into two categories: White/Asian vs. non-Asian minority. 

At all four grades, there were no significant differences overall on pre-test scores among classes 
with different amounts of instruction based on SIE-supported modules. However, there were 
some achievement gaps. Non- Asian minorities had lower pre-test scores on average than their 
White/Asian classmates at each grade level. In addition, females scored slightly lower than 
males on the 4* and 5* grade pre-tests. 

Post-test scores were significantly higher than pre-test scores at all four grade levels. In addition, 
the amount of growth was significantly related to the amount of instructional time based on the 
SIE-supported modules. Table 21 shows the predicted post-test scores (i.e., the expected score 
for a typical student controlling for demographics and other factors included in the analysis) for 
students in classes receiving varying amounts of instructional based on the SIE-supported 
modules. These results show a positive relationship between the amount of instructional time 
based on SIE-supported modules and student achievement, though the effect is relatively small. 



Table 21 

Predicted Post- Test Scores, 



by Amount of Instructional 1 


rime Based on SIE-Supported Modules 




1 Standard Deviation 
Below the Mean Amount of 
Instructional Time 


Mean Amount of 
Instructional Time 


I Standard Deviation 
Above the Mean Amount 
of Instruction Time 


3'^“' Grade 


44.33 


45.58 


46.35 


4“’ Grade 


51.54 


52.09 


52.41 


S'” Grade 


49.10 


51.38 


52.95 


6 "" Grade 


59.00 


61.36 


63.00 



All of the achievement gaps on the pre-test also existed on the post-test. The amount of 
instructional time based on SIE-supported modules was not related to these gaps at either time 
point (i.e., instruction based on the SIE-supported modules neither increased nor decreased the 
achievement gap). 

> Is there a relationship between teacher participation in SIE-provided professional 
development and student achievement? 

HRI also examined whether the extent of teacher participation in SIE-provided professional 
development was related to student achievement. Eor these analyses, HRI used program records 
of teacher attendance at SIE Initial Module Trainings and Module Enrichment sessions to 
classify teachers by the number of trainings they attended related to the topics assessed. Because 
there was limited variation in the professional development attendance at each grade level, HRI 
combined data across the four grade levels to have sufficient statistical power (i.e., the 



Asian students typically outperform all other groups of students, and are often grouped with White students in 
these types of analyses. 
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probability of detecting a difference if one truly exists) of the analysis. As can be seen in Table 
22, the majority of grades 3-6 classes in this study were taught by teachers who had attended an 
Initial Module Training, and almost half were taught by teachers who had attended two or more; 
about one-quarter of the classes were taught by teachers who had attended a Module Enrichment 
session. 



Table 22 



Teacher Participation in SIE Professional Development 





Percent of Classes 
(N = 1,001) 


Number of Initial Module Trainings Attended 




0 


5 


1 


51 


2 or more 


44 


Number of Module Enrichment Trainings Attended 




0 


76 


1 


24 



Students of teachers who had attended two or more Initial Module Trainings had significantly 
greater growth from the pre- to post-test than students of teachers who had attended zero or one 
of these sessions by 0.15 and 0.13 standard deviations, respectively. In addition, students of 
teachers who had attended Module Enrichment gained, on average, 0.06 standard deviations 
more than students of teachers who had not attended. Eigure 9 shows predicted scores at each 
grade level for students of teachers with different levels of attendance at SIE-provided 
professional development. 
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Predicted Test Scores, 

by Teacher Participation in SIE Professionai Deveiopment^ 



3rd Grade 




ME ME ME ME 

□ Pre-Test □ Post-Test 



4th Grade 




ME ME ME ME 

□ Pre-Test □ Post-Test 



5th Grade 



6th Grade 





ME ME ME ME ME ME ME ME 



□ Pre-Test □ Post-Test □Pre-Test □ Post-Test 

* Across all grades, post-test gains of students of teachers attending 2 Initial Module Trainings significantly 
different than students of teachers attending 0 or 1 Initial Module Training, p < 0.05 (HLM). In addition, students 
of teachers attending Module Enrichment had significantly different gains than students of teachers who had not 
attended Module Enrichment, p < 0.05 (HLM). 

Figure 9 



Overall, these results provide additional evidence that the SIE program is having a positive 
impact on students. Teachers are reporting improved student attitudes toward and interest in 
science, and attribute these outcomes to the SIE program. In addition, student learning gains are 
greater in classes where the teachers base more of their instruction on the SIE-supported 
modules. Similarly, greater teacher participation in SIE-provided professional development is 
positively associated with greater learning gains. 
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Summary and Recommendations 



In its third year, the SIE program can be credited with a number of accomplishments. The 
program has brought a third cohort on board and provided a Strategic Planning Institute for those 
schools. In terms of teacher training, SIE has provided three days of professional development 
to Cohort 3 teachers, and has continued to offer professional development to teachers in Cohorts 
1 and 2. The program has also increased the number of science modules being provided to 
schools. In addition, SIE has hosted a Vision Conference for both current and prospective 
schools, provided additional leadership training for a cadre of teacher leaders, and assisted 
schools with conducting their Showcase of Student Eearning. 

By providing these services, SIE has played a key role in helping participating districts make 
progress toward updating their science programs. Evaluation data indicate that participating 
schools have increased the amount of instructional time devoted to science. In addition, SIE has 
helped schools shift their science instruction from a predominantly textbook-/worksheet-based 
program to a more hands-on, inquiry-based one. According to school and district administrators, 
these changes have led to an increased interest and enthusiasm for science among both students 
and teachers. However, although SIE (and the introduction of the science PSSA) has increased 
emphasis on science, administrators in participating schools also indicated that science continues 
to take a back seat to reading and mathematics as those are the subjects that count toward 
Adequate Yearly Progress. 

Overall, the professional development provided by SIE has been well received by teachers and 
principals. Participating teachers reported that the Initial Module Trainings had given them the 
knowledge and skills they need to implement a science module for the first time. The trainings 
have also generated a great deal of enthusiasm among teachers for teaching science, a subject 
many have shied away from in the past. Teachers who have attended Module Enrichment 
Training indicated that these sessions have been helpful in improving their implementation of the 
module. Participants in both types of professional development have had an increase in their 
perceptions of pedagogical preparedness and pedagogical content knowledge. 

Classroom observations of teachers implementing the modules provided additional insight into 
the impacts of the program. The enthusiasm generated in the professional development has 
carried through into the classroom. In all of the observed lessons, students were excitedly 
conducting the hands-on science activities, collecting and recording data and discussing what 
they observed. The classroom observations also revealed that, while the process of improving 
science instruction has begun, teachers need continued support to complete the transformation. 

In particular, teachers would benefit from additional professional development on the science 
content in the module and how to best use the activities to help students engage with and make 
sense of the targeted science ideas. 

In terms of student outcomes, there are multiple pieces of evidence that the SIE program is 
having positive impacts. Teachers reported a high level of student interest in and enjoyment of 
science, and the vast majority of teachers attributed these positive attitudes to the SIE program. 

In addition, analyses of student achievement data found a positive relationship between the 
extent to which teachers base their science instruction on the SIE-supported modules and student 
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learning in science. The analyses also found that, on average, the more SIE-provided 
professional development teachers participate in, the greater the change in their students’ scores 
from the pre- to post-tests. 

To assist the program in achieving its goal of continuous improvement, HRI offers the following 
recommendations : 

> Teachers have expressed concerns about not having the modules for a long enough 
period of time to implement all of the lessons. The program should consider increasing 
the length of time the modules are in the schools. 

By providing instructional materials and professional development to teachers, SIE has been 
successful at initiating substantive changes to the way science is taught in participating schools. 
However, there have been a number of factors that have limited teachers’ use of the modules, 
including not having the modules for a long enough period of time. In fact, the percentage of 
teachers indicating that the length of time they had the module was problematic increased 
significantly from 2007-08 to 2008-09. The program is already planning to implement a new 
module delivery system for the next school year, which will address the problem to some extent. 
It will also be important for the program to take school calendars into account when deciding 
module delivery and pick-up dates. Eor example, teachers reported losing several days of 
instructional time when they had the module due to school holidays and the PSSA 
administration. This issue will be particularly salient for Cohort 1 schools that will be 
implementing a third module next year. 

> Consider ways to support districts, schools, and teachers in developing articulated 
curricula that incorporate the SIE-provided modules and that also ensure that all state 
science standards will he addressed. 

Another reality that schools and teachers have to contend with is how to address all of the state 
science standards when the modules engage students with fewer topics than traditional 
instructional materials. One strategy to address this issue would be for SIE staff to identify 
specific modifications to the modules (e.g., lessons that can be shortened or skipped altogether), 
but would still allow for students to develop the key concepts. Unless the state substantially 
reduces the number of science standards, this issue will continue to confront teachers as they 
attempt to implement the materials and instructional strategies supported by the SIE program. 

> SIE professional development should both model and make explicit for teachers the 
type of science teaching that will most likely result in students learning important 
science ideas. 

Overall, the Initial Module Trainings have been successful at preparing teachers to implement 
the modules at a mechanical level. These trainings have provided teachers with ample 
experiences to engage with the set-up and implementation of the module activities. However, 
classroom observations have found that teachers struggle with providing appropriate sense- 
making opportunities for their students. To help teachers implement the modules more 
purposefully, the program may want to adopt the “learner hat/teacher hat” model. By using this 
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model, the trainings would provide opportunities for teachers to engage with and make sense of 
the science ideas first as learners. Then, the professional development would have them step 
back and consider how they would implement the lesson as teachers. 

> SIE should focus professional development on helping teachers understand each of the 
targeted concepts in the instructional materials and how the instructional activities 
develop those concepts. 

In addition to increasing the emphasis on helping teachers make sense of the science content 
targeted by the modules, the professional development should help teachers understand the 
content storyline of their module. To implement a module purposefully, teachers need to know 
what ideas are intended to be taught by each lesson (both those that are made explicit and those 
that are implicit). In addition, teachers need to understand how each idea is developed through 
the module, so that the activities are implemented purposefully and do not end up being “activity 
for activity’s sake.” The program’s work to develop content frameworks and storylines, and 
integrate these tools into its professional development, are steps in the right direction. Making 
sure that the Module Enrichment Trainings consistently engage teachers with the content 
storyline will be an important next step to continue teachers on the path towards purposeful 
implementation of the modules. 

> As the program continues to grow and evolve, and as new professional development 
providers are brought on board, it should consider ways to codify key elements of its 
work to ensure consistent high-quality implementation. 

The SIE program has built its foundation on the long history and accumulated knowledge of 
ASSET and its staff. A major challenge any organization faces is maintaining its institutional 
knowledge and sharing that knowledge with new team members. One mechanism for preserving 
this knowledge is to create artifacts which codify key elements of the program and its work. The 
program has begun this process by creating content storylines for the modules it supports, which 
illustrate which science ideas are developed in the modules, and how each activity contributes to 
the development of those ideas. Sharing these content storylines with teacher leaders responsible 
for leading Initial Module Trainings will help ensure a consistent focus on the science content in 
those sessions. As these storylines are shared with all teachers participating in the program, one 
would expect a greater focus on the science ideas when the modules are implemented in 
classrooms. 

In addition, updating the facilitation guides for each module’s Initial Module Training to include 
explicit suggestions for when and how to use different strategies such as “learner hat/teacher hat” 
will both develop the knowledge and skills of the teacher leaders and help ensure consistency in 
these workshops. The newly-developed conversation tool would be an excellent vehicle through 
which the program can help the teacher leaders deepen their understanding of these strategies 
and of the design rationale for the professional development. 



Horizon Research, Inc. 



66 



July 2009 




> Develop additional ways to support teacher leaders in their role as science education 
advocates in their districts. 

The ASSET Leadership Conference was successful at building the teacher leaders’ enthusiasm 
for science education improvement. To maintain this momentum, and to support the teacher 
leaders in this role, the SIE program may want to consider ways to provide support to teacher 
leaders throughout the school year. Allowing the teacher leaders to share successes and 
challenges may provide both motivation and ideas for continuing the work. One option for this 
sharing would be to hold a series of face-to-face meetings; another option would be to utilize a 
web-based tool. Although there are pros and cons to both approaches, on-going discussion and 
support would likely be useful to the teacher leaders. 

Another tactic the program may want to consider would be to provide the teacher leaders with 
tools that they could use in their role as science education advocates. Eor example, a number of 
teacher leaders reported that they have been asked to make presentations about the program to 
their school boards. Providing a template for such a presentation (including PowerPoint slides or 
overheads) would reduce the preparation burden on the teacher leaders, help ensure a consistent 
message about the program is being delivered, and increase the likelihood that the teacher 
leaders experience success. 
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Table A-1 

Principals Indicating that Potential Program Services Would be Very Valuable 





Percent of Responding Principals 


Cohort 1 
(N = 50) 


Cohort 2 
(N = 37) 


Cohort 3 
(N=13) 


Ideas and tools for securing the resources needed for science instruction 


65 


51 


58 


Ideas and tools for assessing progress of our school’s science program 


64 


53 


58 


Ideas and tools for creating opportunities for teacher collaboration 


54 


68 


54 


Ideas and tools for observing and providing feedback to teachers about their 








science instruction 


54 


62 


69 


Professional development for me to increase my own understanding of how to 








integrate science with mathematics and literacy 


58 


53 


46 


Ideas and tools for making science more of a priority in the school 


28 


43 


46 


Ideas and tools for increasing parental support/interest/involvement in our 








school’s science program 


46 


44 


38 


Professional development for me to increase my own understanding of 








effective science teaching 


44 


49 


31 


Ideas and tools for leveraging other district resources (e.g., professional 








development days) to support science instruction 


44 


46 


38 


Ideas and tools for increasing community support/interest/involvement in our 








school’s science program 


42 


43 


38 


Professional development for me to increase my own science content 








knowledge 


20 


27 


31 
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Table A-2 

Teacher Opinions of the Quality of SIE Initial Module Training, by Type of Training 





Percent of Responding Teachers 
Agreeing^ 


Initial Module 
Training on 
2”'* Module 
(N = 961) 


Initial Module 
Training on 
1*‘ Module 
(N = 584) 


The facilitators shared tips and suggestions for successfully implementing 






the module. 


97 


97 


The goals of this workshop were clear. 


95 


97 


This workshop familiarized me with the activities and materials in the 






module. 


95 


97 


This workshop increased my understanding of the science content in the 






module. 


95 


96 


The facilitators encouraged active participation and investigation by all 






participants. 


94 


97 


This workshop was relevant to my classroom instruction. 


94 


93 


The various components of this workshop were useful in meeting its goals. 


93 


96 


The facilitators were well prepared. 


93 


95 


Question about the science content in the module were adequately addressed 






by the facilitators. 


93 


95 


This workshop prepared me to use the teaching strategies promoted by SIE. 


93 


95 


This workshop increased my confidence in my ability to teach using the SIE 






module. 


93 


94 


This workshop was worth the time that I invested. 


92 


94 


The facilitators modeled effective teaching strategies. 


91 


92 


The facilitators explicitly discussed how, when and why to use different 






teaching strategies. 


91 


92 


I would recommend this professional development to a colleague. 


90 


94 


This workshop reflected careful planning and organization. 


90 


93 


There were opportunities for participants to express their views and 






collaborate with peers. 


88 


91 


Adequate, time, structure, and guidance were provided for participants to 






discuss the SIE modules and pedagogical strategies with each other. 


80 


82 


Adequate time, structure, and guidance were provided for participants to 






reflect individually on the substance of this workshop. 


78 


83 



Includes those indicating “Agree” and “Strongly Agree” on a five-point scale of 1 “Strongly Disagree” to 5 “Strongly 



Agree.” 



Horizon Research, Inc. 



July 2009 





Table A-3 

Teacher Opinions of the Pace of 
SIE Initial Module Trainings, by Type of Training 





Percent of Responding Teachers 


Initial Module 
Training on 2"'* 
Module 
(N = 933) 


Initial Module 
Training on 
1*‘ Module 
(N = 561) 


Much too slow 


3 


1 


Somewhat too slow 


16 


13 


Appropriate 


62 


66 


Somewhat too fast 


13 


16 


Much too fast 


6 


4 



Table A-4 

Teachers’ Opinions of Factors Affecting 
Their Use of the SIE-Provided Module, by Cohort 





Percent of Responding Teachers 
Agreeing^ 




Cohort I 
(N = 455) 


Cohort 2 
(N = 675) 


Cohort 3 
(N = 
431) 


Factors Facilitating Use 

The training I received from SIE made it easier for me to use the modules. 


95 


94 


91 


Other teachers in my school provided a support system for use of the modules. 


86 


88 


91 


My own science background was helpful when I was teaching from the 
module. 


82 


81 


82 


My SOS facilitated my use of the modules. 


77 


77 


79 


I received assistance from SIE outside of the workshop that helped me to use 
the modules. 


29 


21 


24 


Factors Inhibiting Use 

The pressures to teach mathematics and/or reading inhibited my use of the 
modules. 


53 


52 


43 


The amount of time required to prepare for instruction with the SIE modules 
was problematic. 


52 


45 


37 


I did not have the modules for long enough to use as much of them as I wanted 
to. 


50 


37 


37 


There was not enough instructional time for science to effectively use the SIE 
modules. 


46 


47 


39 


I am not able to cover all of the science topics I am supposed to because of the 
time it took to implement an SIE module. 


42 


44 


39 


The amount of time I was required to be out of the classroom was problematic. 


15 


15 


11 


My lack of experience in science made it more difficult for me to teach from 
the module. 


9 


10 


10 



Includes those indicating “Agree” and “Strongly Agree” on a four-point scale of 1 “Strongly Disagree” to 4 “Strongly 
Agree.” 
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Table A-6 

Principals’ Opinions of Factors Affecting Teachers’ 
Implementation of the SIE-Provided Modules, by Cohort 





Percent of Responding 
Principals Agreeing^ 




Cohort 1 
(N = 50) 


Cohort 2 
(N = 37) 


Cohort 3 
(N=13) 


Factors Facilitating Use 

The training teachers received from SIE made it easier for them to use the 
modules 


98 


100 


100 


Teachers provided each other with a support system for using the modules 


94 


100 


92 


Our Support on Site (SOS) facilitated teachers’ use of the modules 


88 


100 


92 


Teachers received assistance from SIE outside of the workshop that helped 
them to use the modules 


52 


68 


54 


Factors Inhibiting Use 

The amount of time teachers were required to he out of the classroom was 
prohlematic 


38 


44 


38 


The time of year that teachers received their modules was problematic 


50 


27 


23 


There was not enough science instructional time for teachers to complete the 
SIE modules 


40 


35 


38 


Teachers did not have the modules for enough time to complete all of the 
lessons 


46 


35 


8 


The pressures to teach mathematics and/or reading inhibited teachers’ use of the 
modules 


34 


31 


38 


Teachers were not able to cover all of the science topics they were supposed to 
because of the time it took to implement an SIE module 


18 


32 


31 


The amount of time required to prepare for instruction with the SIE modules 
was problematic for teachers 


20 


24 


23 


Too much time elapsed between the module training and module delivery 


20 


19 


8 


Teachers’ lack of science content knowledge made it more difficult for them to 
teach from the modules 


14 


14 


31 


Teachers’ lack of experience in science made it more difficult for them to teach 
from the modules 


12 


8 


31 



Includes those principals agreeing or strongly agreeing with each statement. 
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To facilitate the reporting of large amounts of survey data, and beeause individual questionnaire 
items are potentially unreliable, groups of survey questions that measure similar ideas ean be 
eombined into “eomposites.” Each composite represents an important eonstruet related to 
seienee teaehing or professional development. Cronbaeh’s Coeffieient Alpha is a measure of the 
reliability of a eomposite (i.e., the extent to whieh the items appear to be measuring the same 
eonstruet). A Cronbach’s Alpha of 0.6 is considered aeeeptable, 0.7 fair, 0.8 good, and 0.9 
excellent. 

Eaeh eomposite is ealeulated by summing the responses to the items assoeiated with that 
eomposite and then dividing by the total points possible. In order for the eomposites to be on a 
100-point seale, the lowest response option on eaeh scale was set to 0. As a result, someone who 
marks the lowest point on every item in a eomposite reeeives a seore of 0, and someone who 
marks the highest point on every item reeeives a seore of 100. It also assures that 50 is the true 
mid-point. The denominator for each composite is determined by eomputing the maximum 
possible sum of responses for a series of items and dividing by 100; e.g., a nine-item composite 
where eaeh item is on a seale of 0-4 would have a denominator of 0.36. 



Table B-1 



Composite; Teacher Perceptions of Pedagogical Preparedness 



Preparedness to: 


Post-PD 

(Prior) 


Post-PD 

(Now) 


Use the inquiry-based teaching strategies embedded in the SIE module 


Q8a-p 


Q8a-n 


Use science notebooks to support student learning of the content in the SIE module 


Q8b-p 


Q8b-n 


Use the EERA (Eocus, Explore, Reflect, Apply) and 5 E (Engagement, Exploration, 
Explanation, Elaboration, Evaluation) Learning Cycles to teach using the SIE 
module 


Q8c-p 


Q8c-n 


Manage the logistics of the SIE module 


Q8d-p 


Q8d-n 


Handle classroom management issues with the SIE module 


Q8e-p 


Q8e-n 


Use questioning strategies to elicit student thinking about the science concepts in the 
module. 


Q8f-p 


Q8f-n 


Examine student work to assess student thinking about the science concept in the 
module. 


Q8g-p 


Q8g-n 


Teach the science concepts addressed in the module. 


Q8h-p 


Q8h-n 








Number of Items in Composite 


8 


8 


Reliability (Cronbach’s Coefficient Alpha) 


0.95 


0.94 
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Table B-2 



Composite: Teacher Perceptions of Pedagogical Content Knowledge 



Understanding of: 


Post-PD 

(Prior) 


Post-PD 

(Now) 


Student learning goals (big ideas) in the SIE module 


Q7a-p 


Q7a-n 


Science content in the SIE module at the level that the students are expected to learn 
it 


Q7b-p 


Q7b-n 


Science content in the SIE module at a deeper level than what students are expected 
to learn 


Q7c-p 


Q7c-n 


Ideas (either correct or incorrect) that students are likely to have about the content in 
the SIE module before instruction 


Q7d-p 


Q7d-n 


How the activities in the module connect conceptually with one another. 


Q7e-p 


Q7e-n 


How the activities in the module contribute to understanding the big ideas of the 
module. 


Q7f-p 


Q7f-n 


Real-world connections to the science content in the module. 


Q7g-P 


Q7g-n 








Number of Items in Composite 


7 


7 


Reliability (Cronbacb’s Coefficient Alpha) 


0.95 


0.93 



Table B-3 



Composite: Teacher Perceptions of Principal Support 



My Principal... 


End-of Year 
Questionnaire 


Makes attending the SIE professional development a priority 


Q14a 


Provides opportunities for teachers participating in SIE to meet and share ideas 


Q14b 


Is not very knowledgeable about the SIE program 


Q14c 


Is enthusiastic about the SIE program 


Q14d 


Makes science teaching a priority 


Q14e 


Is supportive of teachers participating in SIE 


Q14f 


Encourages implementation of the SIE modules 


Q14g 


Encourages the use of innovative science instructional strategies 


Q14h 


Accepts the noise associated with the activity based SIE modules 


Q14i 


Encourages teachers to integrate science and literacy 


Q14i 






Number of Items in Composite 


10 


Reliability (Cronbacb’s Coefficient Alpha) 


0.90 
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Table B-4 

Composite; Teacher Perceptions of Student Attitudes 



My students... 


End-of Year 
Questionnaire 


Enjoyed learning about science 


Q15a 


Were interested in science 


Q15b 


Liked science more than previous student I have taught 


Q15c 


Were eager to explore science on their own, outside of the classroom 


Q15d 


Were enthusiastic about science class 


Q15e 


Wanted to learn more science 


Q15f 






Number of Items in Composite 


6 


Reliability (Cronbacb’s Coefficient Alpha) 


0.81 
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Table C-1 

HLM Student Assessment Analysis Results - Module Use 







Regression Coefficients (and standard errors) 






Grade 3 


Grade 4 


Grade 5 


Grade 6 


Intercept 
Cohort ^ 


33.60 


(0.21) 


40.51 


(0.23) 


37.19 


(0.20) 


49.18 


(0.43) 


Cohort 1 


0.39 


(0.45) 


-1.04* 


(0.48) 


-0.37 


(0.58) 


0.12 


(0.99) 


Cohort 3 


0.85 


(0.70) 


0.84 


(0.82) 


2.30* 


(0.87) 


6.27* 


(1.52) 


Number of Topics Taught 


-0.13 


(0.23) 


0.53 


(0.30) 


-0.08 


(0.24) 


-0.76 


(0.54) 


Amount of Time Spent Using SIE Modules 
Amount of Time Spent Using Other 


0.02 


(0.15) 


-0.12 


(0.13) 


-0.11 


(0.18) 


0.34 


(0.30) 


Resources 


-0.41 


(0.29) 


-0.51 


(0.34) 


-0.17 


(0.32) 


-0.46 


(0.58) 


Female 
Cohort ' 


0.17 


(0.31) 


-1.21* 


(0.32) 


-0.71* 


(0.35) 


0.13 


(0.51) 


Cohort 1 


— 


— 


— 


— 


-0.24 


(0.70) 


-0.97 


(1.17) 


Cohort 3 


— 


— 


— 


— 


2.23* 


(1.07) 


-2.47 


(1.77) 


Number of Topics Covered 
Amount of Time Spent Using SIE 


— 


— 


— 


— 


-0.38 


(0.30) 


0.49 


(0.64) 


Modules 

Amount of Time Spent Using Other 


— 


— 


— 


— 


-0.15 


(0.22) 


0.00 


(0.36) 


Resources 


— 


— 


— 


— 


0.14 


(0.39) 


-0.10 


(0.67) 


Non- Asian Minority 


-4.15* 


(0.52) 


-5.65* 


(0.55) 


-5.10* 


(0.60) 


-7.31* 


(0.85) 


English Language Learner 


-1.23 


(1.44) 


-0.20 


(1.59) 


-0.55 


(1.66) 


-1.81 


(2.37) 


Growth from Pre to Post 
Cohort ' 


9.69* 


(0.31) 


10.73* 


(0.26) 


8.74* 


(0.29) 


6.54* 


(0.35) 


Cohort 1 


-0.97 


(0.67) 


-0.46 


(0.54) 


-0.42 


(0.63) 


-0.99 


(0.80) 


Cohort 3 


-0.02 


(1.03) 


-0.61 


(0.92) 


-0.75 


(0.95) 


-0.51 


(1.23) 


Number of Topics Covered 


0.92* 


(0.35) 


-0.53 


(0.34) 


0.60* 


(0.26) 


0.58 


(0.43) 


Hours of Instruction Using SIE Modules 
Hours of Instruction Using Other 


0.61* 


(0.22) 


0.35* 


(0.15) 


1.02* 


(0.20) 


0.73* 


(0.25) 


Materials 


-0.57 


(0.43) 


0.84* 


(0.38) 


-0.66 


(0.35) 


-0.55 


(0.47) 


Female 
Cohort ^ 


0.29 


(0.29) 


0.00 


(0.30) 


0.15 


(0.30) 


0.39 


(0.42) 


Cohort 1 


— 


— 


— 


— 


— 


— 


0.12 


(0.95) 


Cohort 3 


— 


— 


— 


— 


— 


— 


0.69 


(1.45) 


Number of Topics Covered 
Hours of Instruction Using SIE 


— 


— 


— 


— 


— 


— 


0.10 


(0.52) 


Modules 

Hours of Instruction Using Other 


— 


— 


— 


— 


— 


— 


0.32 


(0.30) 


Materials 


— 


— 


— 


— 


— 


— 


-0.58 


(0.55) 


Non- Asian Minority 
Cohort ^ 


-2.20* 


(0.61) 


-1.56* 


(0.60) 


-1.87* 


(0.58) 


0.08 


(0.74) 


Cohort 1 


1.73 


(1.31) 


3.24* 


(1.24) 


0.02 


(1.15) 


1.96 


(1.58) 


Cohort 3 


3.84 


(1.97) 


4.58* 


(2.11) 


-0.44 


(1.80) 


-1.78 


(2.25) 


Number of Topics Covered 
Hours of Instruction Using SIE 


-1.30 


(0.77) 


-1.13 


(0.74) 


0.20 


(0.49) 


-1.78* 


(0.84) 


Modules 

Hours of Instruction Using Other 


0.11 


(0.46) 


0.40 


(0.35) 


-0.25 


(0.36) 


0.61 


(0.43) 


Materials 


0.64 


(0.91) 


0.49 


(0.86) 


0.23 


(0.72) 


0.92 


(0.98) 


English Language Learner 
Cohort ^ 


-1.25 


(1.48) 


-0.65 


(1.72) 


0.49 


(1.82) 


2.75 


(1.96) 


Cohort 1 


-0.85 


(3.14) 


3.26 


(4.03) 


-3.47 


(3.68) 


— 


— 


Cohort 3 


2.78 


(3.65) 


4.41 


(4.72) 


-4.15 


(5.16) 


— 


— 


Number of Topics Covered 
Hours of Instruction Using SIE 


2.18 


(1.55) 


1.74 


(2.20) 


-0.57 


(1.39) 


— 


— 


Modules 

Hours of Instruction Using Other 


0.67 


(1.17) 


0.40 


(1.02) 


0.12 


(1.02) 


— 


— 


Materials 


-0.49 


(2.02) 


-2.43 


(2.16) 


0.21 


(2.11) 


— 


— 



*p < 0.05. 

Note: All variables were grand-mean centered except the Growth from Pre to Post variable which was uncentered. 
’ Versus Cohort 2 schools 



Horizon Research, Inc. 



July 2009 





Table C-2 

HLM Student Assessment Analysis Results - 
Teacher Participation in SIE Professional Development 





Regression Coefficients (and standard errors) 


Intercept 


-0.31 


(0.04) 


Cohort ‘ 






Cohort 1 


-0.08* 


(0.02) 


Cohort 3 


0.10* 


(0.04) 


Grade Level ^ 






Grade 3 


-0.21* 


(0.03) 


Grade 4 


-0.22* 


(0.03) 


Grade 5 


-0.13* 


(0.03) 


Initial Module Training ^ 






1 Initial Module Training 


-0.05 


(0.05) 


2+ Initial Module Trainings 


-0.10* 


(0.05) 


Module Enrichment Training 


0.05 


(0.03) 


Female 


-0.03 


(0.06) 


Cohort ^ 






Cohort 1 


0.00 


(0.03) 


Cohort 3 


0.09 


(0.05) 


Grade Level ^ 






Grade 3 


0.01 


(0.04) 


Grade 4 


-0.08 


(0.04) 


Grade 5 


-0.07 


(0.04) 


Initial Module Training ^ 






1 Initial Module Training 


-0.02 


(0.06) 


2+ Initial Module Trainings 


-0.02 


(0.07) 


Module Enrichment Training 


0.01 


(0.04) 


Non- Asian Minority 


-0.42* 


(0.02) 


English Language Learner 


-0.11 


(0.06) 


Growth from Pre to Post 


0.62* 


(0.05) 


Cohort ‘ 






Cohort 1 


-0.10* 


(0.03) 


Cohort 3 


0.02 


(0.04) 


Grade Level ^ 






Grade 3 


0.43* 


(0.03) 


Grade 4 


0.43* 


(0.03) 


Grade 5 


0.21* 


(0.03) 


Initial Module Training ^ 






1 Initial Module Training 


0.03 


(0.05) 


2+ Initial Module Trainings 


0.15* 


(0.05) 


Module Enrichment Training 


0.06* 


(0.03) 


Female 


0.00 


(0.06) 


Cohort ^ 






Cohort 1 


0.01 


(0.03) 


Module Enrichment Training 


-0.01 


(0.03) 


Grade Level ^ 






Grade 3 


0.02 


(0.04) 


Grade 4 


-0.02 


(0.04) 


Grade 5 


-0.03 


(0.04) 


Initial Module Training ^ 






1 Initial Module Training 


0.01 


(0.06) 


2+ Initial Module Trainings 


0.05 


(0.06) 


Cohort 3 


-0.01 


(0.05) 


Non-Asian Minority 


-0.11 


(0.09) 


Cohort ‘ 






Cohort 1 


0.12* 


(0.05) 


Cohort 3 


0.10 


(0.08) 


Grade Level ^ 






Grade 3 


-0.12 


(0.07) 


Grade 4 


-0.05 


(0.06) 



Horizon Research, Inc. 



July 2009 




Grade 5 


-0.11 


(0.06) 


Initial Module Training ^ 






1 Initial Module Training 


-0.04 


(0.10) 


2+ Initial Module Trainings 


0.07 


(0.10) 


Module Enrichment Training 


-0.05 


(0.07) 


English Language Learner 


0.07 


(0.16) 


Cohort ‘ 






Cohort 1 


0.05 


(0.14) 


Cohort 3 


0.06 


(0.18) 


Grade Level ^ 






Grade 3 


-0.24 


(0.17) 


Grade 4 


-0.27 


(0.17) 


Grade 5 


-0.08 


(0.16) 


Initial Module Training ^ 






1 Initial Module Training 


-0.01 


(0.18) 


2+ Initial Module Trainings 


-0.17 


(0.20) 


Module Enrichment Training 


0.09 


(0.17) 



* p < 0.05. 

Note: All variables were grand-mean centered except the Growth from Pre to Post, Initial 



Module Training, and the Module Enrichment Training variables which were uncentered. 
’ Versus Cohort 2 schools 
^ Versus Grade 6 students 

^ Versus teachers attending no Initial Module Trainings 
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